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I.  EXECUTIVE  SUMMARY 


The  purpose  of  this  paper  is  to  describe  a  process  for  utilizing  existing  data  bases 
and,  if  necessary,  developing  new  ones  to  answer  key  policy  questions  regarding  DoD 
funding  for  various  forms  of  training.  This  paper  presents  a  general  cost-effectiveness 
methodology  for  analyses  that  support  questions  on  the  desirability  of  increasing, 
decreasing,  or  holding  constant  DoD  funding  for  various  training  modalities.  Although  our 
methodology  will  be  demonstrated  in  the  context  of  a  specific  case  study,  it  is  designed  to 
be  applicable  to  a  wide  range  of  training  resource  allocation  problems. 

This  methodology  draws  on  proven  principles  of  training  cost  analysis  put  forth  in 
detail  by  other  authors,  as  well  as  DoD  cost-effectiveness  analysis  methodologies 
developed  and  applied  by  practitioners.  It  is  designed  to  make  optimal  use  of  existing  data 
on  training  cost  and  effectiveness.  The  methodology  emphasizes  simplicity  and  robusmess 
over  detail  and  complexity  and,  with  detailed  refinements  and  augmentatiems,  applies  to  the 
entire  spectrum  of  potential  questions  on  training  cost-effectiveness. 

This  summary  dtscribes  our  conceptual  approach  to  the  identification  and 
assessment  of  cost  and  effectiveness  data  required  for  the  support  of  training-policy 
decisions.  Our  approach  is  based  on  a  requirements-oriented,  functional  point  of  view 
(i.e.,  we  define  th  problem  and  determine  what  we  need  to  fix  it).  The  innovative  aspect 
of  our  approach  is  the  development  of  a  systematic  Training  Funds  Decision  Methodology 
(TFDM),  which  is  demonstrated  by  a  case  study  example.  The  TFDM  is  suggested  for  use 
by  training  policy-makers  in  the  development  and  justification  of  their  funding  requests  to 
Congress  and  sub^uent  resource  allocation  decisions.  Should  this  funding  methodology 
ultimately  prove  to  be  valid,  acceptable,  and  supportable  by  existing  or  planned  data 
repositories,  it  could  be  translated  into  specifications  for  a  decision-support  system. 

The  remainder  of  this  summary  describes  briefly  the  need  for  a  TFDM,  the  overall 
purpose  of  such  a  methodology,  the  specific  obj«:tives  that  must  be  achieved  to  establish  a 
valid  TFDM,  a  description  of  the  process  employed  in  its  development,  and  conclusions 
and  recommendations. 
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A .  PROJECT  GOALS  AND  OBJECTIVES 


1.  Goals 

The  project  goal  is  to  "assist  the  Training  Policy  Directorate,  OASD(FM&P)  in 
assessing  the  adequacy  of  training  cost  data  bases  in  support  of  OSD  and  the  Services  and 
their  utility  in  supporting  training  policy  decisions"  (IDA  Task  Order  No.  T-L2-407  of 
23  May  86).  To  accomplish  this  goal,  the  IDA  team  examined  the  macro-level  training 
funds  data  base  at  the  Training  and  Performance  Data  Center  (TPDC)  to  determine  its 
completeness,  relevance,  and  utility  in  tlie  development  of  training  policy  by  using 
hypothetical  funding  decisions.  Based  on  these  findings,  the  task  was  modified  to  include 
case  studies  of  selected  funding  decisions  as  a  means  of  testing  the  cost  and  benefit  analysis 
capabilities  inherent  in  the  TPDC  data  base.  Additionally,  we  identified  the  cost  and 
effectiveness  data  and  data  bases  necessary  to  assist  OSD  and  the  Services  in  formulation 
and  support  of  training  policy  decisions  at  the  DoD  level. 

Since  the  focus  of  this  study  was  the  development  of  data  requirements  for  training 
decisions  rather  than  the  evaluation  of  an  existing  data  c^al^ity,  we  developed  a  functional 
architecture  for  a  decision  methodology.  This  methodology  will  support  "top-down" 
training  policy  development  by  employing  requirements  derived  from  the  tasks  to  be  trained 
as  criteria  for  identifying  and  evaluating  data  elements  frcmi  existing  resources. 

2 .  Technical  Objectives 

The  objectives  of  the  task  are  threefold: 

•  Analyze  the  training  policy-making  (decision)  function 

•  Develop  a  methodology  for  cost-effectiveness  analysis  in  support  of  the 
training  policy-making  function 

•  Demonstrate  the  methodology  via  a  case  study. 

a.  Objective  1:  Analyze  the  Training  Policy-Making  Function 

An  analysis  of  the  training  policy-making  function  was  conducted  on  the  basis  of 
personnel  interviews  with  training  managers  and  the  mission  statements  for  the  Directorate 
of  Training  Policy  (DTP)  within  OASD(FM&P),  We  began  with  a  concise  statement  of  the 
mission  and  functional  responsibilities  of  the  DTP.  Next,  the  role  of  a  DoD  training  policy¬ 
maker  (macro-level  training  manager)  was  defined  and  used  to  deduce  the  content  and  level 
of  aggregation  requirements  for  training  data  and  the  decision-making  tools  to  support  him. 
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From  this  analysis  we  developed  an  understanding  of  the  process  to  be  supported 
by  the  TFDM.  First,  we  reviewed  and  analyzed  the  Training  Management  Model  (TMM) 
for  the  conversion  of  resources  to  training  products  and  training  management.  Second,  we 
examined  relevant  aspects  of  the  Canadian  Training  Management  Model  (CTMM).  These 
included  information  requirements,  decision  contexts,  and  efficiency  in  cost-effectiveness 
analysis.  Next,  we  investigated  the  application  of  a  previously  proposed  cost  element 
structure  (CES)  for  training  programs,  courses,  and  devices.  Finally,  performance 
requirements  for  the  training  policy-making  function  were  analyzed  to  guide  the 
development  of  a  comprehensive  TFDM. 

b.  Objective  2:  Develop  a  Methodology  for  Cost-Effectiveness 
Analysis  to  Support  the  Training  Policy-Maker 

In  support  of  this  objective,  we  reviewed  a  number  of  previously  reported 
methodologies  for  and  approaches  to  cost-effectiveness  analysis  of  military  training 
options.  We  also  identified  in  some  detail  the  cost  and  effectiveness  data  requirements  and 
data  sources  for  cost-effectiveness  analysis  of  training.  Finally,  based  on  the  functional 
requirements  from  Objective  1  and  the  candidate  methods  explored  in  Objective  2,  we 
developed  a  training  funds  decision  methodology  (TFDM)  to  support  the  DoD-level 
training  policy-maker. 

c.  Objective  3:  Demonstrate  the  TFDM  Using  a  Sample  Case  Study 

To  demonstrate  the  application  of  the  IFDM  we  conducted  a  sample  case  study  in 
which  we  addressed  the  following  question:  "Should  support  for  computer-based 
instructionAnteractive  videodisc  technology  (CBl/IVD)  for  electronics  maintenance  training 
be  increased  by  a  specified  amount?"  The  product  of  this  scenario-driven  case  study  was  a 
preliminary  evaluation  of  the  TFDM  concept,  which  included  diagnostic  information  for 
each  major  step  in  the  process.  This  modular  philosophy  was  adopted  so  that  fixes  could 
be  made  to  any  major  step  of  the  TFDM  without  having  to  dismantle  or  discard  the  entire 
concept.  This  low-risk  approach  to  decision  support  methodology  design  is  based  on  the 
concept  of  rapid  prototyping.  As  such,  constructive  criticism  will  continually  upgrade  the 
effectiveness  of  the  methodology  under  development 

B.  CONCLUSIONS  AND  RECOMMENDATIONS 

The  utility  of  the  TFDM  lies  in  its  inherent  objectivity  and  reliability.  The  TFDM 
will  provide  the  training  policy-maker  with  a  systematic,  auditable,  data  driven 
methodology  that  is  comprehensive  enough  to  serve  his  current  functions  and  flexible 
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enough  to  accommodate  planning,  programming,  and  budgeting  changes.  The  TFDM  will 
provide  the  training  policy-maker  with  a  set  of  flexible,  comprehensive  rules  to  guide  his 
decision  process  concerning  the  type,  amount,  medium,  and  mix  of  training  that  is  best  for 
the  military  today.  These  rules  will  help  assure  that  policy  justifications  are  derived  from 
cost  and  effectiveness  data  as  well  as  past  experience.  They  will  also  guide  the  policy¬ 
maker  to  appropriate,  existing  training  cost  and  effectiveness  data  bases.  If  the  required 
data  or  data  bases  do  not  exist,  the  TFDM  will  assist  the  training  policy-maker  in  specifying 
the  design  characteristics  of  the  necessary  infOTmation  repositories.  In  theory,  justifications 
based  on  a  rigorous  logic  trail  and  reliable  data  should  be  more  defensible  than  those  based 
on  subjective  preferences. 


Another  advantage  of  this  decision  support  methodology  is  the  ease  with  which 
people  can  learn  to  operate  it.  Since  its  processes  are  objective,  repeatable,  and  open  to 
scrutiny,  the  TFDM  and  its  outputs  can  be  applied  to  other  spheres  of  interest.  It  appears, 
for  example,  that  the  concepts  underlying  the  TFDM  may  be  transferable  to  other  DoD 
policy-making  activities.  Finally,  and  possibly  most  important,  the  TFDM  can  reduce  the 
costs  and  time  associated  with  the  development  of  training  policy.  On  the  basis  of  these 
conclusions,  we  recommend  that  ASD(FM&P)TP  do  the  following: 

•  Continue  to  develop  and  refine  the  TFDM  moving  toward  system  design  and 
implementation. 


•  Begin  the  development  of  TFDM  data  bases  using  past  cost-effectiveness 
analyses. 

•  Perform  additional  detailed  case  studies  to  validate  TFDM  and  associated  data 

base  design  and  development  concepts.  • 

•  Establish  an  organization  within  OASD(FM&P)TP  to  oversee  and  guide  the 
collection  of  the  data  necessary  to  conduct  military  training  cost-effectiveness 
analyses. 


C.  REPORT  OVERVIEW 

In  Chapter  n  we  analyze  the  training  policy-making  function.  In  Chapter  HI,  we 
lay  the  groundwork  for  our  methodology  for  cost-effectiveness  analysis  in  support  of  the 
training  policy-making  function,  the  TFDM,  and  then  present  that  methodology.  In 
Chapter  FV,  we  present  a  sample  case  study  in  which  the  TFDM  is  applied  to  a  training 
decision  involving  the  application  of  CBI/IVD  technology  to  electronics  maintenance 
training.  Finally,  in  Chapter  V,  we  provide  an  overall  summary,  draw  conclusions,  and 
make  recommendations. 
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n.  ESTABLISHMENT  OF  REQUIREMENTS  FOR  DESIGN  OF 
A  TRAINING  FUNDS  DECISION  METHODOLOGY  (TFDM) 


This  chapter  reports  on  the  analytic  processes  used  to  develop  TFDM  design 
requirements.  The  first  subsection  provides  an  analysis  of  the  training  policy-making 
function  and  describes  the  mission  and  functional  responsibilities  of  the  Directorate  of 
Training  Policy.  Next,  the  role  of  the  training  policy-maker  is  discussed,  followed  by  a 
statement  of  the  policy-maker's  need  for  high-level  training  data  bases  and  tools. 

The  second  and  third  subsections  explain  the  processes  underlying  the  TFDM.  A 
model  for  converting  resources  to  training  products  and  the  Training  Management  Model 
are  presented  and  discussed.  Then,  some  relevant  aspects  of  the  Canadian  Training 
Management  Model  are  presented,  including  its  information  requirements,  decision 
contexts,  and  efficiency  in  cost-effectiveness  analysis.  Finally,  the  application  of  a 
previously  proposed  training  cost-element  structure  is  explained. 

In  the  final  two  subsections  we  present  a  summary  of  our  analysis  and  die  resulting 
functional  requirements  for  guiding  the  development  of  the  TFDM. 

A.  ANALYSIS  OF  THE  TRAINING  POLICY-MAKING  FUNCTION 

1 .  The  Mission  and  Functional  Responsibilities  of  the  Directorate  of 
Training  Policy  (DTP) 

The  Directorate  of  Training  Policy  (DTP)  of  the  Office  of  the  Assistant  Secretary  of 
Defense  for  Force  Management  and  Personnel  [OASD(FM&P)]  is  responsible  for  the 
development  of  military  training  policy  for  the  DoD.  The  DTP  is  responsible  for  reviewing 
Service  training  policies,  programs,  and  budgets  for  effectiveness,  efficiency,  and 
consistency  with  DoD  policy.  DTP  is  also  responsible  for  providing  justification  of 
training  programs  and  budgets  to  Congress.  DTP'S  functional  areas  of  responsibility 
include  all  training  of  military  individuals,  crews,  and  units,  to  include  recruit,  specialized 
skill,  flight,  professional,  and  on-the-job  (OTJ)  training  and  operational  unit  training  and 
exercises. 
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Within  this  mission,  DTP  has  eight  primary  functional  responsibilities  and  four 
enabling  responsibilities.  The  primary  functional  responsibilities  are  as  follows: 

( 1 )  Development  of  policy  for  planning  and  conducting  military  training. 

(2)  Development,  acquisition,  and  use  of  simulators  and  training  devices. 

(3)  Development  of  guidelines  for  computer-based  instruction. 

(4)  Production  of  the  annual  Military  Manpower  Training  Report  (MMTR). 

(5)  Support  of  the  Deputy  Assistant  Secretary  of  Defense  (Military  Manpower  and 
Personnel  Policy)  [DASD  (MM&PP)]  as  chairman  of  the  OSD  Steering 
Committee  on  Training  and  Training  Technology. 

(6)  Represent  the  OASD  (FM&P)  on  the  Euro-NATO  Training  Group. 

(7)  Perform  liaison  with  National  Security  Industrial  Association  (NSIA)  and 
American  Defense  Preparedness  Association  (ADPA)  activities. 

(8)  With  the  Office  of  the  Deputy  Under  Secretary  of  Defense  (Research  and 
Advanced  Technology),  co-sponsor  and  evaluate  Joint-Service  prototype 
training  devices  and  technology. 

The  four  additional  responsibilities  are: 

( 1 )  Review  Service  training  policies/programs/budgets. 

(2)  Provide  justification  for  training  programsAesources  to  Congress. 

(3)  Manage  the  training-related  research  program  of  ASD(FM&P). 

(4)  Respond  to  General  Accounting  Office/DoD  Inspector  General  (GAO/DoDIG) 
reports  on  training  matters. 

Accomplishment  of  the  mission  and  functional  responsibilities  is  mediated  by 
preserving  essential  training  resources  and  enhancing  training  effectiveness  through 
simulation/technology.  Other  important  concerns  include,  but  are  not  limited  to,  the 
operations  of  the  Training  Performance  Data  Center  (TPDC)  and  portability  of  computer- 
based  instructionAnteractive  videodisc  courseware. 

2.  Role  of  the  Training  Policy*Maker 

The  role  of  the  DoD-level  training  policy-maker  is  to  respond  to  approved  national 
defense  needs  by  providing  adequately  trained  personnel  on  time  and  for  an  acceptable 
cost.  The  training  policy-maker  is  positioned  at  the  highest  level  of  the  DoD  training 


management  organization.  From  this  position  he  manipulates  the  military  training  system 
to  meet  current  personnel  needs  generated  by  continual  analysis  of  global  threats.  This  is 
done  through  the  judicious  use  of  budgetary  powers  to  approve,  support,  and  defend  or,  as 
required,  disapprove  and  oppose  the  proposed  budgets  for  particular  parts  of  the  training 
system.  The  training  manager's  power  to  influence  the  budget  is  constrained  only  by  the 
need  to  respond  to  training  requirements  imposed  by  ertemal  forces,  such  as  weapon 
system  development  and  personnel  policies,  and  the  need  to  respond  in  a  timely  and  cost- 
effective  manner. 

Although  training  is  critical  to  the  proper  functioning  of  the  military,  advocates 
frequently  have  difficulty  securing  resources  in  a  highly  competitive  financial  arena  because 
it  is  very  difficult  to  predict  quantitatively  the  benefits  of  training.  Hence,  the  training 
policy-maker  needs  to  be  able  to  estimate  accurately  the  impact  of  training  policy  decisions 
on  readiness  and  force  effectiveness.  If  the  relationship  between  the  amount  and  type  of 
training  and  performance  in  combat  can  be  quantified  reliably,  the  formulation  of  training 
policy  will  be  less  subjective  and  the  training  manager's  ability  to  compete  for  resources 
will  be  improved. 

3.  Need  for  Policy-Level  Training  Data  Bases  and  Tools  for  Their  Use 

As  one  analyst  noted,  in  recognition  of  the  need  to  improve  the  validity  and 
supportability  of  training  resource  decisions, 

...[a]  1982  Defense  Science  Board  (DSB)  study  group  recommended  that 
DOD  set  up  a  centralized  data  and  analysis  center  for  the  defense  training 
community.  The  DSB  concluded  that,  while  good  training  data  appeared  to 
exist,  the  information  was  not  always  in  a  form  best  suited  to  the  needs  of 
training  managers.  (Sicilia,  1985) 

As  a  result,  the  Training  and  Performance  Data  Center  (TPDC)  was  established  and 
has  become  a  repository  for  existing  data  bases  and  tools  of  potential  value  to  the  training 
policy  decision-maker.  In  addition,  new  data  bases  and  analytical  software  to  facilitate  their 
use  are  currently  under  development.  However,  the  utility  of  existing  and  planned  data 
repositories  to  the  training  policy-maker  is  not  presently  clear.  This  is  in  part  because  the 
data  bases  and  analytical  tools  have  not  been  developed  with  a  top-down  view  toward 
supporting  the  particular  needs  of  the  DoD  training  policy-maker.  In  addition,  the  TPDC 
has  had  to  serve  the  needs  of  a  diverse  group  of  users,  many  of  whom  have  little  interest  in 
the  data  requirements  and  analysis  needs  of  the  DoD-level  training  decision-makers. 
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In  recognition  of  this  need,  EDA's  Macro-Level  Training  Task  was  designed  to: 

(1)  Provide  an  overview  of  the  kinds  of  data  available  and  their  relevance  to 
training  management  issues  and  to  the  development  of  macro-level  training  data 
bases. 

(2)  Identify  the  data  required  to  evaluate  several  illustrative  and  representative 
policy-making  situations. 

(3)  Collect  and  evaluate  the  data  required  for  prototype  policy-making  situations  to 
demonstrate  the  range  of  policy  decisions  that  can  be  supported  by  the  Macro- 
Level  Cost  Data  Bases. 

The  present  paper,  together  with  two  previous  reports,  was  produced  to  fulfill  these 
objectives.  The  first  report  was  entitled,  Macro-Level  Training  Data:  A  Conceptual  Model 
Relating  Training  Cost  and  Effectiveness  Data  to  Training  Policy  Interests  and  Issues 
(Gibson  and  Orlansky,  1988).  It  developed  a  conceptual  model  of  the  training  system  from 
the  training  manager's  perspective.  Based  on  the  model  and  a  summary  of  the  available 
data,  some  strategies  and  options  for  developing  the  required  data  bases  were  presented. 

The  second  paper  in  this  series  entitled  Macro-level  Training  Case  Study-Computer 
Based  Instruction/Interactive  Videodisc  (CBHIVD)  Data  System-Phase  I  Report,  (Gibson 
et  al.,  1989)  documented  the  first  phase  of  the  analysis.  Its  goal  was  to  determine  the  types 
of  data  required  and  to  assess  the  adequacy/availability  of  these  data  using  a  case  study 
approach.  The  study  team  developed  a  representative  training  policy  question,  "Should 
support  for  computer  based  instructionMdeodisc  technology  ((TBI/IVD)  be  increased  or 
decreased  and,  if  so,  by  what  amount?"  and  attempted  to  answer  it  using  available  data. 
This  effort  addressed  the  second  and  third  analytical  objectives  above. 

The  present  report  emphasizing  a  "rapid  prototyping"  approach  to  training  policy 
development  is  designed  to  do  the  following: 

(1)  Analyze  and  understand  the  training  policy  decision-making  function, 
including  its  data  requirements  and  analytical  process  needs. 

(2)  Develop  a  concise  statement  of  a  Training  Funds  Decision  Methodology  that  is 
the  practical  embodiment  of  the  training  policy  decision-making  function 
analyzed  above. 

(3)  Conduct  two  versions  of  a  case  study  which  seek  to  apply  the  methodology  as 
deHned  above. 

(4)  Draw  conclusions  concerning  the  data  needs  of  the  training  funds  decision 
process  vis-^-vis  existing  data  and  currently  planned  data  bases. 


B .  EFFORTS  TO  MODEL  THE  MANAGEMENT  OF  TRAINING 


1 .  A  Model  for  Conversion  of  Resources  to  Training  Products 

As  discussed  earlier,  the  DoD  training  manager  manipulates  the  military  training 
system  by  converting  an  objective,  tangible  quantity  (money)  into  a  relatively  subjective, 
intangible  quality  (combat  readiness).  From  the  DoD  training  manager's  point  of  view, 
money  is  the  common  denominator  for  support  of  the  various  components  of  a  training 
system.  It  represents  training  resources  such  as  student  and  instructor  time,  training 
facilities,  and  equipment.  The  process  of  converting  resources  into  training  activities  and 
products  is  schematically  represented  in  Figure  II-l.  As  part  of  the  process,  the  training 
system  produces  voluminous  data  related  to  the  consumption  of  resources  and  the 
production  of  trained  personnel. 

Gibson  and  Orlansky  (1988)  identified  a  clear  link  between  the  requirements 
generated  by  technical/organizational  systems  and  the  resulting  training  products  that 
should  make  it  possible  to  use  the  training  data  bases  for  formulating  and  supporting  policy 
decisions  in  a  precise  and  quantitative  manner.  To  help  identify  the  data  critical  to 
management  of  the  training  policy-making  process,  Gibson  and  Orlansky  developed  a 
Training  Management  Model  (TMM).  This  model  assumes  that  resources  used  for  training 
(such  as  aircraft,  ships,  tanks,  or  weapons)  are  not  available  for  other  Service  needs.  The 
fundamental  principles  of  the  TMM  are  elaborated  below. 

2.  The  Training  Management  Model 

The  Training  Management  Model  (TMM)  is  discussed  in  terms  of  its  basic 
dynamics  and  components,  including  its  cost-production  function,  performance 
effectiveness  function,  and  readiness  evaluation  function. 

a.  Basic  Dynamics 

The  underlying  assumption  of  the  TMM  is  that  the  military  training  system  is  driven 
by  requirements.  The  requirements  in  turn  are  based  on  national  strategy,  perceived  threat, 
the  character  and  needs  of  present  and  projected  platforms  and  weapon  systems,  and  the 
quantity,  quality,  and  turbulence  of  personnel.  Figure  11-2  shows  the  relations  between 
those  requirements  and  resources,  training  activities,  performance,  evaluation/quality 
control,  and  readiness  status.  This  complex  of  external  and  internal  factors  serves  to 
establish  the  anxrunt  of  trairting  needed  and  the  conqx>sition  of  the  training  system. 


Source:  Gibson  and  Orlansky,  1988. 


Figure  11-1.  The  Conversion  of  Resources  into  Training  Activities  and  Products 
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Source:  Gibson  and  Odansky,  1988. 

Figure  ll•2.  Training  Management  Model 
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Training  resources  are  allocated,  and  the  needed  individual  and  collective  training 
programs  are  conducted  to  maintain  the  quality  and  quantity  of  trained  personnel 
established  by  the  requirements.  The  results  of  the  training  provide  feedback  which  also 
serves  to  modify  the  requirements. 

Training  requirements  are  determined  by  estimating  the  probable  future 

attrition  of  required  levels  and  skill  mix  of  personnel  due  to  completion  of 

enlistments,  retirements,  and  the  need  for  increases  in  personnel  levels  due 

to  increases  and/or  changes  in  weapon  systems  and  force  structures. 

(Solomon,  1986) 

b.  Structural  Components  of  the  Model 

The  DoD  training  manager  must  be  concerned  with  three  major  functions  addressed 
by  the  training  system:  the  cost  and  production  aspects  of  training;  the  effectiveness  of 
training;  and  the  level  of  combat  readiness  resulting  from  training. 

These  three  major  training  management  functions  in  the  TMM  (Fig.  11-3)  represent 
alternative  ways  that  the  DoD  training  manager  interacts  with  the  training  system.  The 
basic  functioning  of  the  overall  training  system  does  not  change  as  the  manager's 
orientation  and  data  needs  change  to  respond  to  various  speciHc  problems. 

Cost-Production  Function.  The  Cost-Production  function  is  a  subset  of  the 
TMM  elements  that  describe  the  normal  or  baseline  operation  of  the  training  system.  This 
is  represented  in  Figure  n-3a  by  a  continuous  feedback  loop  consisting  of  the  following 
four  elements:  Requirements,  Resources,  Individual  Training,  and  Collective  Training. 
Based  upon  approved  sets  of  requirements,  resources  in  the  form  of  budgets,  facilities,  and 
personnel  are  channeled  into  individual  and  group  training  programs  which  ultimately  fulfill 
or  modify  the  existing  requirements. 

The  Cost-Production  function  is  primarily  quantitative  in  nature.  It  responds  to  the 
training  manager's  concern  with  costs  and  the  ability  of  the  training  system  to  produce  the 
numbers  of  trained  personnel  needed  to  meet  approved  organizational  requirements  on 
schedule  and  within  budget  This  is  the  standard  operating  condition  of  the  training  system 
except  when  it  fails  to  provide  an  acceptable  product  in  tenns  of  the  quantity  and  quality  of 
trained  personnel.  The  bulk  of  the  training  manager's  information  requests  will  be 
generated  from  within  this  functional  module  of  training  system  operation. 

The  ability  of  the  training  manager  to  operate  in  the  Cost-Production  function  of  the 
TMM  depends  upon  two  assumptions: 
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Source:  Gibson  and  Orlansky,  1988. 

Figure  11-3.  Training  ManagMnent  Model  Functions 
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•  That  the  training  courses  and  events  are  effective  in  conveying  the  necessary 
information  and  skills  to  the  personnel  involved,  and 

•  That  trained  personnel,  teams,  and  units  advance  the  state  of  readiness  and 
potential  combat  effectiveness  of  the  armed  forces. 

When  these  assumptions  have  questionable  validity,  the  training  manager  needs  to 
validate  the  assumptions  empirically  with  appropriate  performance  or  readiness  data. 

Performance  Effectiveness  Function.  The  Performance  Effectiveness 
function  of  the  TMM  supplements  the  Cost-Production  function  in  two  ways.  Within 
training  courses,  individual  performance  data  (e.g.,  test  scores)  support  the  training  quality 
control  process  and  provide  the  basis  for  deciding  whether  an  individual  should  continue 
the  next  phase  of  training,  gra'^aate,  recycle,  or  leave  the  program.  Data  on  group 
performance  in  weapon  systems  (training  or  operational)  provide  a  basis  for  evaluating  the 
effectiveness  of  training  courses  or  techniques. 

The  Performance  Effectiveness  function  in  Rgure  II-3b  consists  of  Requirements, 
Resources,  Individual  Traiiung,  Collective  Training,  and  Performance  Evaluation/C^ality 
Control.  There  is  continuous  feedback  between  the  Individual  Training  element  and  the 
Performance  Evaluation/Quality  Control  element  However,  (xily  an  intermittent  feedback 
is  represented  between  Collective  Training  and  the  Performance  Evaluation/CJuality  Control 
element.  This  function  is  concerned  primarily  with  the  ability  of  the  individual  trainees  to 
perform  well  on  the  job.  Therefore,  it  provides  data  on  the  quality  of  training  or,  more 
specifically,  on  the  quality  of  the  performance  skills  of  trained  personnel. 

Data  on  performance  effectiveness  (test  scores  or  job  performance  evaluations)  can 
have  a  dramatic  impact  on  training  policy;  however,  most  of  these  data  needs  are 
formulated  and  the  resulting  data  are  collected  and  used  at  the  individual  training  (micro-) 
level  where  they  have  minimal  direct  impact  on  policy-level  decision-making.  A  positive 
relationship  between  training  and  readiness  and  potential  combat  effectiveness  is  implicitly 
assumed.  When  this  link  is  questioned,  the  training  manager  needs  to  collect  readiness 
information  to  validate  this  assun^tion. 

Readiness  Evaluation  Function.  This  third  function  of  TMM  operation  looks 
at  the  quantitative  and  qualitative  effects  of  training  on  combat  readiness.  Readiness  criteria 
can  range  from  a  certification  that  the  approved  training  requirements  have  been  met  to  an 
actual  measure  of  combat-relevant  performance.  The  outcomes  from  these  operations  are 
capable  of  having  a  significant  impact  on  the  operation  of  the  training  system.  However, 


once  a  problem  has  been  resolved  satisfactorily,  the  training  system  goes  back  to  the 
baseline  operating  condition  represented  by  the  Cost-Production  function. 

The  Readiness  Evaluation  function,  like  the  Performance  Evaluation  function, 
estimates  the  collective  performance  capability  of  operational  units  and  can  operate  in  two 
distinct  ways.  The  first  is  represented  in  Figure  n-3c  by  Requirements,  Resources,  and 
Readiness.  These  three  elements  provide  continuous  evaluation  of  how  well  the 
prescriptive  readiness  requirements  have  been  met,  i.e.,  have  all  the  individuals  and 
functional  units  completed  the  required  training  courses  and  events?  The  second  way  of 
functioning  uses  all  the  elements  in  Figure  II-3c  and  provides  a  possible  link  between 
training  and  empirical  readiness  evaluations. 

c .  Summary  of  the  TMM 

The  TMM  prov'ides  a  conceptual  model  of  the  major  interactions  between  the  DoD 
training  policy-maker  and  the  military  training  system  with  respect  to  the  related  issues  of 
cost,  effectiveness,  and  readiness  due  to  training.  Based  on  a  small  set  of  assumptions 
about  the  function  and  goals  of  the  training  policy-maker  and  the  nature  of  military  training, 
the  TMM  provides  a  broad  perspective  of  the  types  of  data  most  likely  to  be  required  to 
resolve  training-policy  questions. 

All  functions  of  Cost  Production,  Performance  Effectiveness,  and  Readiness 
Evaluation  may  operate  at  the  same  time.  The  ones  that  are  emphasized  depend  on  what 
training  issues  are  dominant  at  that  time.  The  Cost-Production  function  is  the  normal 
operating  condition  Here,  the  DoD  training  manager  is  primarily  concerned  with  costs,  the 
effectiveness  of  the  training  system  in  providing  the  mix  of  trained  personnel  required,  and 
ways  to  minimize  the  cost  of  providing  trained  personnel. 

If  the  effectiveness  or  cost-effectiveness  of  a  specific  training  effort  becomes 
questionable,  then  the  performance  effectiveness  of  the  training  system  becomes  the 
dominant  concern.  Here,  performance  data,  primarily  on  individuals  or  groups  in  training 
or  recently  graduated  from  training,  should  be  examined  by  the  training  manager.  If  the 
contribution  of  training  to  combat  readiness  is  questionable,  then  the  readiness  evaluation 
aspect  of  training  becomes  dominant  In  this  case,  the  focus  of  the  manager's  interest  may 
be  either  the  adequacy  of  the  training  system  in  providing  all  the  courses  and  unit  training 
events  required  for  combat  readiness  or  how  well  the  courses  and  unit  training  events 
contributed  to  some  measure  of  combat  effectiveness.  When  questions  concerning 


pcrfonnance  effectiveness  or  readiness  evaluation  have  been  answered,  the  emphasis 
reverts  back  to  the  cost-production  aspect  of  training. 

From  the  perspective  of  the  DoD  training  policy-maker,  the  most  interesting  data 
will  be  what  resource  expenditure  on  a  particular  training  approach  is  likely  to  produce  a 
5»pecific,  measurable  increase  in  performance.  To  achieve  these  data,  it  will  be  necessary  to 
precisely  quantify  performance  gains  resulting  from  the  application  of  a  training  resource. 
Therefore,  for  the  purpose  of  this  report,  the  essential  features  of  the  TMM  are  its  cost- 
production  and  performance  effectiveness  functions. 

3 .  The  Canadian  Training  Management  Model 

In  November  1987,  the  Bureau  of  Management  Consulting  for  Supply  and 
Services,  Canada,  prepared  a  draft  report  for  the  Directorate  of  Research  and  Development 
(Human  Performance)  of  the  Department  of  National  Defense  entitled  "Issues  in  Cost- 
Effectiveness  Analysis  of  Military  Training."  The  report  included  discussions  of 
information  linkages  in  cost-effectiveness  analysis  of  training,  the  context  of  the  training 
decisions  that  the  cost-effectiveness  analysis  is  meant  to  support,  and  the  evolution  of  cost- 
effectiveness  data  for  training  decisions. 

a.  Information  Linkages 

The  key  elements  and  information  flows  involved  in  the  Canadian  cost- 
effectiveness  analysis  of  training  are  illustrated  in  Figure  11-4.  First,  performance 
objectives  are  derived  from  an  analysis  of  tl»  requirements  of  the  job  or  the  tasks  for  which 
the  training  is  to  be  conducted.  Alternative  training  approaches  are  then  formulated  to  meet 
the  specified  objectives.  The  plans  may  vary  with  respect  to  such  factors  as  the  method  of 
delivery  (e.g.,  self-paced  versus  group),  the  duration  of  training,  location  of  training  (e.g., 
school  versus  on-the-job),  and  the  type  of  equipment  used  (e.g.,  simulation  versus  actual 
equipment).  The  resources  required  to  implement  each  alternative  are  identified  and  costed. 
This  will  normally  require  the  use  of  a  standard  cost  model  to  provide  a  fair  comparison  of 
the  costs  of  uie  various  alternatives.  The  cost  information  is  then  supplied  to  the  decision¬ 
maker.  Therefore,  this  portion  of  the  Canadian  model  is  much  like  the  Cost-Production 
functitm  of  the  TMM. 

The  various  outcomes  of  the  alternative  training  plans  are  then  determined  and 
assessed  with  reference  to  the  training  objectives.  These  outoxnes  are  usually  a  measure  of 


Source:  BMCSS  Canada,  1987. 


Figura  IM.  Information  Linkages  in  Cost«Effectlvenesa  Analysis  of  Training 


the  ability  of  the  trainee  to  perform  specified  tasks  or  to  demonstrate  required  knowledge 
after  completion  of  the  training.  The  outcome  assessments  may  be  quantitative  or 
qualitative,  depending  upon  the  nature  of  the  task  involved.  Quantitative  assessments 
require  direct  measurement  of  outcomes,  usually  with  respect  to  some  specified  standard  of 
performance.  Qualitative  assessments  require  the  use  of  experienced  umpires  or  observers. 
The  objectivity  of  qualitative  assessment  can  be  enhanced,  and  the  effects  of  individual 
bias  minimized,  if  clearly  defined  evaluation  procedures  and  criteria  are  established  first. 
Table  11- 1  provides  a  general  indication  of  the  types  of  tasks  for  which  quantitative 
assessment  may  be  possible  and  contrasts  them  with  tasks  for  which  some  degree  of 
qualitative  assessment  is  likely  to  be  unavoidable.  This  portion  of  the  Canadian  model 
mimics  the  Performance  Effectiveness  Function  of  the  TMM.  The  Canadian  model  does 
not  distinguish  between  performance  and  readiness  as  concepts  requiring  separate 
evaluation. 


Table  11-1.  Relationship  of  Task  Characteristics  to  Method  of 
Training  Outcome  Assessment 


Method  of  Assessment 

Quantitative 

Qualitative 

Many  individuai  tasks 

Many  collective  tasks 

Well-structured  tasks 

Poorly  structured  tasks 

Tasks  undertaken  in  a  specified  and 
controlled  environment 

Tasks  involving  the  ability  to  respond  to  an 
uncertain  environmental  stimulus 

Basic  procedural  tasks 

Tasks  requiring  considerable  cognitive  skill 

Source:  BMCSS  Canada,  1987. 


The  assessment  information,  both  quantitative  and  qualitative,  may  be  supplied  to 
the  decision-maker  who  is  then  in  a  position  to  determine  which  training  plan  is  the  most 
cost  effective.  This  cost-effectiveness  determination  may  be  quite  complex,  especially  if 
the  decision-maker  is  faced  with  a  large  array  of  outcome  assessments  for  each  alternative. 
However,  just  as  resource  costs  may  be  rationalized  through  the  use  of  cost  models,  so 
elements  of  effectiveness  can  be  rationalized  through  the  use  of  formal  effectiveness 
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models.  The  Canadian  model  makes  the  role  and  function  of  the  decision-maker  more 
explicit  than  the  TMM. 

b.  Decision  Context 

The  authors  note  that  in  any  cost-effectiveness  analysis  of  military  training,  the  cost 
elements  and  effectiveness  measures  considered  will  depend  largely  on  the  context  of  the 
training  decision  which  the  analysis  is  intended  to  support.  In  Table  11-2,  the  authors 
present  a  tentative  general  framework  to  illustrate  this  point.  The  cost  elements  are 
presented  in  a  highly  aggregated  form,  and  effectiveness  is  identified  in  terms  of  levels 
(related,  in  part,  to  the  hierarchy  presented  in  Figure  11-5),  rather  than  specific  measures. 
An  actual  analysis  would  require  the  identification  of  a  much  more  detailed  list  of  cost 
elements  within  the  categories  indicated  and  a  range  of  specific  effectiveness  measures  at 
the  levels  indicated. 

In  Table  11-2,  nine  decision  contexts  or  situations  have  been  selected.  Other 
possible  contexts  could  have  been  included  by  the  authors,  but  the  nine  chosen  represent  a 
broad  range  of  options  with  respect  to  the  cost  elements  and  effectiveness  levels 
considered.  In  each  case,  those  cost  elements  that  are  required  for  the  analysis  are  indicated 
with  an  "X".  In  some  decision  contexts,  effectiveness  is  also  allowed  to  vary  and  may  be 
considered  endogenous.  In  other  cases,  the  focus  of  the  decision  is  on  finding  the  most 
efficient  way  of  reaching  a  fixed  effectiveness  standard  [indicated  with  an  "(X)"].  An 
endogenous  variable  is  one  which  is  allowed  to  vary  in  the  analysis  in  response  to  the 
decision  being  made.  An  exogenous  variable,  on  the  other  hand,  is  either  assumed  to  be 
uninfluenced  by  the  decision  or  to  be  beyond  the  scope  of  the  analysis.  Each  of  the  nine 
decision  contexts  is  discussed  below. 

1 .  Methodological  Evaluation.  The  evaluation  of  a  new  method  of  instruction  for 
teaching  a  given  course  in  a  given  location  would  probably  be  concerned  only 
with  initial  investment  and  direct  operating  costs.  Since  the  method  would 
already  have  been  developed,  no  R&D  would  be  required,  while  support  and 
infrastructure  costs  would  be  common  to  all  alternatives  being  considered.  It  is 
likely  that  the  decision  would  be  based  on  relative  efficiency  in  attaining  the 
existing  performance  objectives  of  the  course. 

2.  Instructional  Development.  The  second  case  involves  a  broadening  of  the 
decision  context  in  two  ways.  First,  since  the  decision  involves  the 
development  as  well  as  the  implementation  of  a  new  method  of  instruction,  the 
R&D  costs  would  have  to  be  taken  into  account  Second,  if  the  new  method 
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Table  11*2.  Relationship  Between  Decision  Context,  Cost  Elements,  and 

Effectiveness  Levels 


Decision  Context 


Direct  Operating 


Support 


Training  Infrastructure 


Training  Research 


Specific 


General 


EFFECTIVENESS  LEVELS 
Attainment  of  Course  POs 


Individual  on-the-job  skill 
(with  existing  equipment)  _ 


man/macnine  penormance 
(new  or  modified  equipment) 


ective  unit  performance 


(X) 

(X) 

m 

■ 

■ 

■ 

■ 

\mm 

H 

■ 

Equipment  readiness 


Overall  military  preparedness 


X  >  endogenous  variable 
(X)  -  exogenous  variable  used  as  reference  point 


Source:  BMCSS  Canada,  1987. 
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Levels 

Performance 

Criterion 

Some  Additional  Factors  Affecting 
Performance  at  this  Level 

1. 

Extent  of  irvlividual  attainment 
of  POs  at  end  of  course 

•  individual  characteristics 

•  previous  training 

Increasing 
difficulty  in 
determining 

2. 

Individual  performance  wtthin 

•  aspects  of  unit  environment  affecting 

context  of  unit 

transfer  of  training,  skill  enhancement, 
and  skill  decay 

training 
impact  on 
performance 

3. 

Ability  of  unit  to  accomplish 
mission 

•  manpower 

•  abilities  of  other  unit  members 

•  equipment  characteristics 

•  motivation/cohesion 

•  tactics 

4. 

Ability  of  command  to 
accomplish  mission 

«  abilities  of  other  units 

•  logistics 

•  strategy 

Source:  BMCSS  Canada,  1987. 

Figure  11-5.  Impact  of  Specific  Training  Course  on  Performance 

at  Various  Levels 


involves  different  methods  of  instructional  delivery  (e.g.,  remotely  delivered 
versus  centrally  delivered  classroom  instruction),  then  support  costs  may  well 
be  affected. 

3 .  Simulator  Evaluation.  The  evaluation  of  a  previously  developed  simulator  will 
involve  consideration  of  investment  and  operating  costs,  buL  since  the  decision 
may  involve  comparisons  with  training  on  actual  equipment,  support  costs  will 
also  allow  for  the  possibility  that  the  simulator  might  be  more  effective  than 
actual  equipment  in  training  certain  tasks  (e.g.,  tasks  which,  if  performed  with 
actual  equipment,  would  be  too  dangerous  or  too  expensive).  Thus,  in  this 
case,  the  effectiveness  measures  are  treated  endogenously. 

4.  Occupational  Analysis.  An  occupational  analysis  might  not  be  concerned  with 
cost  at  all.  Such  an  analysis  would  focus  on  the  determination  of  tasks  and 
associated  performance  objectives  dictated  by  tm-the-job  requirements. 

5.  Organizational  Evaluation.  An  evaluation  of  the  way  training  is  organized 
would  focus  on  the  costs  of  the  training  infrastructure  and  the  extent  to  which  it 
meets  job  requirements.  In  such  an  analysis  the  costs  of  individual  courses 
would  probably  be  considered  cxily  in  aggregate. 
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6.  Training  Research.  An  analysis  of  research  into  specific  training  problems 
would  be  concerned  primarily  with  specific  training  research  costs  and  the 
extent  to  which  such  research  might  lead  to  improvement  in  existing  on-the-job 
skills. 

7.  Equipment  Design.  One  aspect  of  the  design  of  new  weapon  systems  (and 
other  equipment)  is  the  ease  with  which  personnel  can  be  trained  to  cq)erate  and 
maintain  the  system.  A  cost-effectiveness  analysis  in  support  of  such  design 
decisions  would  be  likely  to  involve  all  aspects  of  training  delivery  and  the 
determination  of  appropriate  performance  objectives  for  the  tasks  identified. 
Overall  effectiveness  in  this  case  would  be  determined  at  the  level  of  combined 
performance  of  man  and  machine  in  order  to  identify  appropriate  trade-offs 
between  design  and  training  aspects. 

8.  Trade-Offs.  At  a  more  general  level,  decisions  might  be  made  regarding  the 
appropriate  mix  of  training  and  operational  resources  required  to  enable  a  unit 
to  achieve  a  given  level  of  effectiveness.  As  with  the  previous  case,  all 
elements  of  training  course  costs  would  be  involved.  However,  the  focus  of 
effectiveness  assessment  would  be  at  the  unit  level. 

9.  Maintenance  Training  Assessment.  Finally,  an  overall  assessment  of 
maintenance  training  would  be  concerned  mostly  with  infrastructure  costs  and 
training  research  specific  to  maintenance  problems.  Individual  course  costs 
would  probably  be  considered  only  in  aggregate.  The  overall  determinants  of 
maintenance  training  effectiveness  would  probably  be  measures  associated 
with  equipment  readiness.  It  is  likely,  however,  that  such  readiness  would  be 
linked  to  on-the-job  skills  and  associated  training  performance  objectives. 
Finally,  the  authors  note  that  the  major  cost  involved  in  conducting  cost- 
effectiveness  analysis  is  in  acquiring  and  processing  the  data  involved. 

C.  EFFICIENCY  IN  COST-EFFECTIVENESS  ANALYSIS 

Not  only  is  the  Canadian  approach  to  cost-effectiveness  analysis  markedly  similar 
to  our  TMM,  but  the  problems  anticipated  and  encountered  in  its  application  are  virtually 
identical.  The  Canadians  suggest  that  such  problems  may  be  minimi 7i»d,  and  the  analytical 
process  made  more  efficient,  by  ensuring  that  information  and  results  obtained  from  the 
model  are  understandable  and  accessible  immediately  to  the  decision-maker.  In  this  way, 
duplication  of  effort  is  avoided  and  learning  from  experience  is  promoted. 

The  Canadians  concluded  that  the  most  effective  way  of  preserving  model  output 
information  would  be  to  set  up  a  comprehensive  data  base  on  the  costs  and  effectiveness  of 
training.  Figure  11-6  illustrates  the  mechanism  through  which  they  believe  such  a  data  base 
might  evolve.  First,  the  information  flows  fOT  immediate  training-related  decisions  are 
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illustrated.  Front-end  analysis  is  conducted  to  determine  the  task  structure  of  the  job  for 
which  training  is  required  and  to  define  appropriate  performance  objectives.  This 
information  is  then  used  in  conjunction  with  predictive  and  prescriptive  cost  and 
effectiveness  models  and,  where  available,  information  fiom  comparable  systems  in  order 
to  design  the  course  and  select  appropriate  media  for  instruction.  Formative  evaluation  of 
the  course  during  development  may  lead  to  design  modifications.  Finally,  as  experience  is 
gained  with  the  course  in  full  operation,  further  design  modifications  may  occur. 

As  we  have  learned  in  the  United  States,  the  information  generated  by  this  process 
should  not  be  lost.  Following  implementation,  the  Canadians  suggest  that  cost  and 
effectiveness  data  gathered  and  analyzed  for  summative  evaluation  of  the  course  should  be 
fed  into  their  proposed  comprehensive  data  base.  If  information  from  comparable  systems 
was  used  in  the  design  of  the  system,  then  they  suggest  that  this  too  should  be  made  part 
of  the  data  base.  In  addition,  the  data  base  can  be  further  enhanced  by  including 
information  from  other  sources,  such  as  research  on  generic  training  issues.  As  illustrated 
in  Figure  11-6,  it  will,  among  other  things,  facilitate  the  refinement  of  predictive  cost- 
effectiveness  models  and  the  enhancement  of  performance  measures.  The  net  result  should 
be  that  more  effective  use  is  made  of  the  training  budget  and  of  available  training  time. 

To  implement  and  maintain  such  a  repository,  von  Baeycr  (1985)  has  suggested 
establishing  a  central  and  impartial  organization  to  conduct  training  cost-effectiveness 
analyses.  The  concept  of  such  an  impartial  evaluative  body  has  not  been  seriously 
proposed  in  the  United  States.  Here,  the  analyses  themselves  seem  to  be  left  primarily  in 
the  hands  of  those  concerned  with  the  training  issues  and  only  the  data  bases  have  received 
serious  attention  in  terms  of  their  organization  in  a  central  repository,  TPDC. 

The  Canadian  model  lends  credence  to  our  own  TMM.  However,  its  authors  have 
judiciously  left  the  complex  issue  of  readiness  out  of  the  model,  focusing  primarily  on 
performance  as  a  measure  of  training  effectiveness.  As  previously  noted,  they  have  more 
explicitly  represented  the  role  and  functions  of  the  training  resource  manager  (policy- 
makCT)  and  made  more  explicit  reference  to  the  options  that  must  be  evaluated.  They  have 
explicitly  provided  us  with  nine  different  decision  contexts  and  briefly  analyzed  the  varying 
requirements  of  each  for  cost  elements  and  effectiveness  levels  (measures)  in  analyzing 
each.  These  will  prove  valuable  in  our  case  study.  Finally,  the  Canadians  have  suggested 
a  mechanism  for  predicting  the  evolution  of  cost-effectiveness  data  (and  models)  for 
training  decisions,  something  that  should  no  doubt  be  our  ultimate  goal  as  well. 
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Information  from 
Comparable 
Systems 


Front-End 

Analysis 


Predictive  and 
Prescriptive  Cost 
and  Effectiveness 
Models 


Media  Selection  and  Design 
of  Instruction,  Including 
Specification  of  Performance 
Criteria  Measures  and 
Instruments 


Developmental  Fielding  of 
Training  Prototype  and 
Formative  Evaluation 
of  Same 


Full  Implementation  and 
Systematic  Collection  and 
Analysis  of  Cost  and 
Effectiveness  Data 


Comprehensive  Data  Base 
on  the  Cost  and  Effectiveness 
of  Training 


Increasing 
availability 
of  empirical 
data  will  lead 
to  the  redesign 
and/or 
recalibration 
of  the  C-E 
models  and 
to  the 
refinement 
of  perfonnance 
measures 


Information  from 
Other  Sources 
(e.g.,  Research) 


Information  flow  for 
immediate  training 
decisions 


Information  flow  for 
long-run  enhancement 
of  training  decisions 


Source:  BMCSS  Canada,  1987. 


Figure  11*6.  Evolution  of  Cost>Effectlveness  Data  for  Training  Decisions 
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D .  APPLICATION  OF  A  PROPOSED  COST-ELEMENT  STRUCTURE  TO 
TRAINING  PROGRAMS,  COURSES,  AND  DEVICES 

A  careful  consideration  of  all  potentially  useful  elements  of  training  cost  is  critical 
for  proper  and  optimal  performance  of  the  train^^ig  policy-making  function  and  will  be 
essential  in  our  later  specification  and  trial  application  of  the  Training  Funds  Decision 
Methodology.  Knapp  and  Orlansky  (1985)  prepared  Table  n-3  to  illustrate  the  applicability 
of  each  cost  element  in  their  previously  developed  cost  element  structure  ((IIES)  for  defense 
training  including  academic,  maintenance,  and  flight  training.  Each  type  of  training  was 
evaluated  in  previous  studies  (i.e.,  the  cost-effectiveness  of  flight  simulators,  computer 
based  instruction,  and  maintenance  simulators).  Discrimination  between  military  and 
civilian  personnel,  although  an  important  characteristic  of  the  CES,  was  omitted  from 
Table  II-3  because  it  was  not  essential  to  illustrate  its  general  applicability.  The 
applicability  of  each  cost  element  to  each  type  of  training  was  then  estimated  as  indicated  in 
the  legend  appended  to  Table  II-3. 

Knapp  and  Orlansky  found  that  some  instructional  functions  and  the  costs 
associated  with  them  are  essential  to  all  training  systems  regardless  of  their  size, 
development  status,  or  complexity.  Therefore  certain  costs  would  always  be  incurred. 
These  costs  include  program/course/device  (p/c/d)  design,  test  and  evaluation, 
system/project  management,  instructional  materials,  initial  training  of  instructors,  base 
operations,  and  management/support  personnel. 

Additional  functions  and  resources,  specific  to  new  training  programs,  courses,  or 
devices  are  also  required.  For  example,  initial  investment  costs  associated  with  hardware 
and  software  production  apply  to  simulators  and  aircraft  designed  exclusively  for  training, 
but  they  are  not  applicable  to  conventional  and  individualized  instruction  in  which 
computers  are  not  utilized. 

Although  the  assessment  of  degree  of  applicability  in  Table  11-3  was  partially 
judgmental,  the  table  shows  that  a  common  cost  element  structure  can  be  applied  to  the 
broad  range  of  training  programs,  courses,  and  devices.  As  noted  at  the  beginning  of  this 
subsection,  we  will  need  to  consider  carefully  all  potentially  useful  elements  of  training  cost 
for  inclusion  in  the  TFDM.  Knapp  and  (Orlansky  (1983, 1985)  and  the  Canadian  Bureau  of 
Management  Consulting  (1987)  have  presented  comprehensive  cost  element  structures  that 
serve  this  purpose.  Once  we  have  combined  these  with  training  effectiveness  data,  in  cost- 
effectiveness  analyses,  we  should  be  well  on  our  way  to  prescribing  a  comprehensive 
TFDM. 
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Table  11*3.  Illustrative  Application  of  the  Comprehensive  Cost  Element 
Structure  for  Military  Training  Programs,  Courses,  and  Devices 


Academic  Training 


Cost  Categories/Elements 


RESEARCH  AND  DEVELOPMENT 

1.  Design 

2.  Component  Development 

3.  Producibility  Engineering 
and  Planning 

4.  Tooling 

5.  Prototype  Manufacturing 

6.  Data 

a.  Martagerial 

b.  Technical 

7.  P/C/D  Test  and  Evaluation 

8.  System/Project  Management 

9.  Facilities 

10.  Other 

INITIAL  INVESTMENT 
1.  Production 
a.  Nonrecurrin 


b.  Recurring 

(1)  Manufacturirtg 

(2)  Sustaining  Engirwerirtg 

(3)  Sustaining  Tooling 

(4)  Quality  Assurance 

(5)  Other 

c.  Initial  Spates  and  Repair  Parte 


5.  Data 

a.  Managerial 

b.  Technical 

c.  Instruction  Materials 


10.  Initial  Training 

a.  Instructors 

b.  Maintenance  Personnel 

11.  Transportation 

a.  First  Destination 

b.  Second  Destination 

12.  Other 

OPERATING  AND  SUPPORT 
1 .  Direct  Costs 

a.  Instructiortal  Costs 

(1)  Pay  and  Allowances 
(a)  Instructors 

Supervisors, 
AdminiBtrative  and 
Support  Personrtel 
(c)  Maintenance  Personrtel 

(2)  Other  Government 
Personnel  Coste 

(3)  Cortsumption 

(a)  POL 

(b)  Training  Mimitions 

(c)  unities 


(8)  Other 

b.  Training  Activity  Costs 
(1)  Pay  and  Allowances 


Table  11-3.  (continued) 


Cost  Categories/Elements 

Academic  Trainirrg 

Maintenance 

Training 

Right  Training 

Converv 

tional 

Individ¬ 

ualized 

Computer-Based  ‘ 

Actual 

Equip¬ 

ment 

Aircraft 

Simu¬ 

lator® 

Computer- 

Assisted 

Computer- 

Maitaged 

Simu- 
lator  * 

rrainer^ 

Opera¬ 

tional*^ 

No  Computers 

N/ 

N/ 

c.  Airfield  and  Carrier 

Operations  Costs 

(1)  Pay  and  Allowances 

(2)  Other  Government 

Personnel  Costs 

a 

a 

(3)  Other 

a 

a 

d.  Student  Costs 

(1 )  Pay  artd  Allowances 

4 

4 

4 

4 

4 

4 

4 

4 

4 

(2)  Other  Student  Costs 

4 

4 

4 

4 

4 

4 

4 

4 

4 

e.  Other  Direct  Costs 

a 

« 

• 

a 

a 

a 

a 

2.  Indirect  Costs 

a.  Base  Operations 

(1)  Pay  and  Allowances 

4 

4 

4 

4 

4 

4 

4 

4 

4 

(2)  OUwr  Government 

Personrtel  Costs 

4 

4 

4 

4 

4 

4 

4 

4 

4 

• 

• 

a 

a 

a 

• 

a 

d.  Command  Support  Costs 

— 

n 

n 

im 

(1 )  Pay  and  Allowances 

(2)  Other  Government 

Personrtel  Costs 

(3)  Other 

e.  Other  Irrdirect  Costs 

Bi 

IHii 

a 

Assumes  new  harcAvare  and/or  software. 

D 

Designed,  produced,  and  operated  as  a  trainer. 

c 

Essentially  operational  configuration  and  performance.  May  be  used  in  primary  miseion  role  but  used  as  trainer. 

♦  Always  applicable. 

•  Maybeapplicable,  depeiaiing  on  the  context  of  the  problem  presented,  assumptions,  ground  njles  for  the  analysis. 

and  characteristics  of  the  training  program/course/device  of  tnteresL 

(Blank)  Not  applicabie. 

Source:  Knapp  and  Orlansky,  1985. 


E.  REQUIREMENTS  FOR  A  TRAINING  FUNDS  DECISION 
METHODOLOGY 

As  we  have  seen,  the  DoD  training  manager  reviews  Service  training  policies, 
programs,  and  budgets  for  effectiveness,  efficiency,  and  consistency  with  DoD  policy.  On 
the  basis  of  this  review,  the  training  policy-maker  provides  supporting  justification  for 
training  programs  and  budgets  to  Congress. 

In  the  next  section,  we  develop  a  prototype  training  funds  decision  methodology 
(TFDM),  based  upon  the  mission  and  responsibilities  of  the  DTP  and  the  role  of  the 
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training  policy-maker  as  examined  in  this  chapter.  We  begin  with  our  concept  of  a  training 
management  model  (TMM)  as  the  core  of  this  methodology,  supplemented  with  our 
knowledge  of  the  relevance  of  various  elements  of  cost  and  training  effectiveness  to 
different  types  of  training  designed  to  meet  varying  objectives.  The  formulation  of  our 
methodology  was  guided  by  our  knowledge  of  the  state  of  the  art  of  military  (as  well  as 
nonmilitary)  cost-effectiveness  analysis,  as  it  applies  to  the  requirements  we  are  seeking  to 
support.  Ultimately,  we  will  wish  to  specify  our  methodology  in  terms  of  process  flow 
diagrams,  but  we  can  begin  here  with  the  flow  of  functional  requirements  for  the  TFDM. 

In  later  sections  of  this  report,  we  apply  our  methodology  to  two  versions  of  a  case 
study,  analyze  the  data  needs  of  the  methodology  vis-k-vis  available  data  bases,  and 
suggest  future  directions  for  work,  including  the  development  of  an  automated  computer 
tool  to  support  the  TFDM. 

Based  on  our  analysis,  the  TFDM  will  be  required  to  perform  five  functions  which 
can  be  described  briefly  as  follows. 

1 .  Frame  the  Question.  Based  upon  the  general  concerns  being  raised  in  the 
specific  training  funds  decision  context,  the  TFDM  must  match  the  current 
analysis  situation  with  one  template  or  schema  chosen  from  a  predefined 
taxonomy  of  possible  analysis  situations. 

2 .  Make  the  Options  Explicit.  Based  upon  specific  questions  and  issues, 
including  training  or  performance  objectives,  the  TFDM  must  develop 
alternative  training  plans.  This  may  require  some  up-hront  analysis  to  ensure 
adherence  to  cost  and  performance  constraints. 

3 .  Determine  Costs  of  Options.  The  TFDM  must  estimate  the  costs  of  each 
training  option  in  all  relevant  cost  categories.  This  may  require  the  use  of  a 
standard  cost  model  to  provide  a  fair  comparison  of  the  costs  of  the  various 
alternatives. 

4 .  Determine  the  Training  Effectiveness  of  Each  Option.  The  TFDM 
must  help  to  predict  job  performance  productivity/proficiency  gains  compared 
to  present  practice  that  would  result  from  each  training  option.  In  some  cases, 
the  research  literature  may  be  our  best  and  only  source  for  such  data. 

5.  Perform  Cost-Effectiveness  Calculations  and  Draw  Conclusions. 
The  TFDM  must  determine  the  productivity/pronciency  gains  to  be  expected 
per  unit  cost  for  each  training  option  and  provide  conclusions  regarding  the 
most  desirable  option,  taking  apprqjriate  constraints  (c.g.,  budgetary,  time  to 
train)  into  account 


F.  SUMMARY 


A  number  of  authors  have  attempted  to  capture  the  essential  elements  of  the  policy¬ 
making  function  for  military  training  as  performed  by  the  DoD-level  training  manager.  In 
this  section,  we  define  the  role  of  the  DoD-level  training  manager  and  address  the  need  for 
macro-level  training  data  bases  and  tools  for  their  use,  after  providing  a  succinct  statement 
of  the  mission  and  functional  responsibilities  of  the  DTP  of  the  OASD(FM&P)TP. 

Based  on  the  firmly  identified  need  for  macro-level  training  data  bases  and 
techniques  for  their  use,  we  laid  the  foundation  for  this  effort  by  clarifying  its  objectives 
and  products.  Since  we  belive  that  policy  is  made  as  the  DoD-level  training  manager 
converts  money  into  combat  readiness,  we  presented  the  TMM  to  show  how  the  training 
manager  does  this  conversion  process.  The  essential  components  of  this  model  are 
requirements,  resources,  training  activities,  performance  evaluation,  quality  control,  and 
readiness  status.  The  structural  components  are  the  cost-production  function,  the 
performance  effectiveness  function,  and  the  readiness  evaluation  function. 

Next,  we  reviewed  the  work  of  the  Canadian  Bureau  of  Management  Consulting  to 
augment  the  work  of  Gibson  and  Orlansky  (1988)  and  Gibson  et  al.  (1989).  Specifically, 
we  looked  at  the  information  linkages  in  cost-effectiveness  analysis  of  training,  which 
essentially  provided  us  with  a  Canadian  version  of  our  TMM  with  the  role  of  the  decision¬ 
maker  in  evaluating  alternatives  made  more  explicit  and  the  concept  of  readiness  not 
represented  in  the  Canadian  model.  We  went  on  to  focus  on  the  varying  nature  of  the 
macro-level  training  decision  contexts  and  how  it  affects  the  level  of  our  analyses  of  both 
cost  and  effectiveness  elements.  Finally,  we  examined  the  Canadians'  views  on  providing 
for  eff  tiency  in  cost-effectiveness  analysis.  They  take  what  we  in  the  United  States  have 
attempted  at  TPDC  a  step  further,  proposing  data  bases  for  integrated  cost-effectiveness 
analysis  (not  independent  cost  and  effectiveness  data  bases)  and  an  independent  cost- 
effectiveness  evaluation  organization  to  perform  all  required  military  cost-effectiveness 
analyses  as  an  impartial  body. 

Finally,  we  returned  to  a  consideration  of  the  cost  data  elements  we  must  draw 
upon  and  how  they  are  conditioned  for  the  type  of  training  under  consideration.  Careful 
consideration  of  the  appropriate  cost  categories  and  elements  will  be  critical  for  the  success 
of  the  training  funds  decision  process  and  the  supporting  training  funds  decision 
methodology. 
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m.  A  METHODOLOGY  FOR  COST-EFFECTIVENESS 
ANALYSIS  IN  SUPPORT  OF  TRAINING  POLICY 


In  this  section  we  describe  a  methodology  for  cost-effectiveness  analysis  in  support 
of  policy-making  for  training.  First,  we  review  a  number  of  previously  reported 
methodologies  for  cost-effectiveness  analysis  of  military  training  options.  Next,  we 
identify  in  some  detail  the  training  cost  and  effectiveness  data  requirements  and  data 
sources  for  cost-effectiveness  analysis  of  military  training  options.  Finally,  drawing  upon 
earlier  material,  we  present  our  own  Training  Funds  Decision  Methodology  (TFDM)  in 
support  of  the  DoD-level  policy-maker  for  training. 

A.  PREVIOUS  METHODOLOGIES  FOR  COST-EFFECTIVENESS 
ANALYSIS  OF  MILITARY  TRAINING 

In  this  subsection  we  review  seven  previously  published  studies,  all  of  which  focus 
on  the  use  of  methodologies  for  cost-effectiveness  analysis  of  military  training.  Several  of 
these  studies  are  themselves  lengthy  reviews  of  other  cost-effectiveness  studies.  The 
purpose  here  is  to  lay  both  a  theoretical  and  pragmatic  foundation  for  what  is  to  follow, 
based  upon  a  recent  history  of  such  analyses. 

1 .  Cost  and  Effectiveness  of  Computer  Based  Instruction  (Orlansky  and 
String,  1979) 

Orlansky  and  String  (1979)  evaluated  the  cost  and  effectiveness  of  computer  based 
instruction  (CBI)  for  military  training  on  the  basis  of  30  studies  conducted  since  1968. 
Computer  assisted  instruction  (CAI)  and  computer  managed  instruction  (CMI)  were  found, 
on  the  average,  to  save  approximately  30  percent  of  the  time  required  by  students  to  train 
conventionally.  Cost  savings  attributed  to  CAI  and  CMI  in  the  various  studies  were  based 
on  estimates  of  pa,  and  allowances  of  students  for  the  time  saved  by  these  methods  of 
instruction.  The  authors  found  that  allowances  were  seldom  made  for  the  costs  of  the  CAI 
or  CMI  equipment  and  courseware,  instructors,  and  other  costs  incremental  to  computer 
based  instruction.  No  cost-effectiveness  analyses  could  be  performed  by  the  authors  for 
lack  of  the  proper  data,  even  though  many  of  the  original  studies  had  purponed  to  support 
the  cost-effectiveness  of  computer  based  instructicm. 


2.  Cost-Effectiveness  and  Maintenance  Simulators  (Orlansky  and  String, 
1981) 

Orlansky  and  String  (1981)  evaluated  the  cost-effectiveness  of  maintenance 
simulators,  compared  to  actual  equipment  trainers,  for  training  military  maintenance 
technicians.  They  first  sought  to  establish  the  effectiveness  of  maintenance  simulators 
versus  actual  equipment  trainers  when  measured  by  student  achievement  at  school  and 
according  to  supervisors'  ratings  of  the  job  performance  of  students  trained  either  way. 
They  found  no  significant  differences  in  training  effectiveness.  Next,  they  sought  to 
establish  the  relative  acquisition  costs  of  the  two  alternatives.  They  found  that  the 
acquisition  cost  of  maintenance  simulators  is  typically  less  and  that  the  life  cycle  cost  of 
maintenance  simulators  over  a  15-year  period  would  cost  40-60  percent  as  much  as  actual 
equipment  trainers,  according  to  a  life-cycle  cost  comparison.  Based  on  these  data,  they 
concluded  that  maintenance  simulators  are  a  cost-effective  alternative  compared  with  actual 
equipment  trainers. 

3.  Framework  for  Evaluating  the  Cost-Effectiveness  of  Military 
Equipment  and  Procedures  (Chatelier  et  al.,  1985) 

Chatelier,  Harvey,  and  Orlansky  (1985)  provided  a  framework  for  evaluating  the 
cost-effectiveness  of  equipment  and  procedures  used  in  military  training.  In  their  work 
they  suggested  the  use  of  the  Transfer  Effectiveness  Ratio  (TER)  proposed  by  Roscoe 
(1971,  1972).  The  TER  equals  the  time  required  to  reach  criterion  performance  using 
actual  equipment  without  prior  use  of  a  simulator  (A),  minus  the  time  required  to  reach 
criterion  performance  using  actual  equipment  with  prior  use  of  a  simulator  (AsubS), 
divided  by  the  time  spent  training  in  the  simulatOT  (S)  [TER  =  (A-AsubS)/S].  A  TER  of 
greater  than  one  indicates  that  total  training  time  (actual  time  plus  simulator  time)  decreases 
for  the  tasks  practiced  on  the  simulator.  They  also  supported  the  use  of  the  Training  Cbst 
Ratio  (TCR)  which  is  the  ratio  of  the  cost  of  the  new  training  approach  (simulator)  to  the 
cost  of  the  existing  training  approach  (actual  equipment).  A  TC31  of  less  than  one  says  that 
the  new  training  approach  (simulator)  is  cheaper  than  the  old  one. 

Finally,  they  used  the  (Tost  Effectiveness  Ratio  (CER),  i.e.,  the  TCIR  divided  by  the 
TER.  If  this  ratio  is  less  than  one,  then  the  new  approach  is  more  cost-effective.  For 
example,  Chatelier  et  al.  (1985)  report  that  flight  simulators  save  about  50  percent  of  the 
time  needed  by  pilots  to  train  in  aircraft  and  cost  only  12  percent  as  much  to  use.  This 
means  a  CER  of  0.24,  which  says  that  one  additional  unit  of  proficiency  achieved  with 
simulators  was  only  24  percent  as  costly  as  one  achieved  with  the  actual  equipment.  It  is 

in-2 


clear  that  flight  simulators  are  extremely  cost-effective,  compared  to  the  use  of  aircraft  alone 
for  training. 

4.  Cost-Effectiveness  of  Military  Training  (Orlansky,  1985) 

In  a  review  of  studies  addressing  the  cost-effectiveness  of  military  training, 
Qrlansky  (1985)  concluded  that  flight  simulators,  CBI,  and  maintenance  training  simulators 
are  as  effective  as  aircraft,  conventional  classroom  instruction,  and  actual  equipment, 
respectively.  Data  from  approximately  100  studies  were  reported  to  support  this  finding. 
Orlansky  noted  that  these  methods  of  training  save  student  time,  investment  costs,  and 
operating  costs  in  the  case  of  flight  and  maintenance  simulators.  Credible  cost  data  were 
not  available  for  CBI  at  that  time.  Orlansky  used  the  arguments  of  equal  effectiveness  at 
less  cost  and  greater  effectiveness  at  equal  cost  to  support  the  claim  of  greater  cost- 
effectiveness  of  CBI  and  simulators. 

5.  Model  of  Cost/Effectiveness  Analysis  of  Training  Devices  (von  Baeyer, 
1985) 

Von  Baeyer  (1985)  presents  a  very  comprehensive  model  for  cost-effectiveness 
analysis  of  training  devices.  The  model's  goal  is  to  enable  the  training  procurement  officer 
to  predict,  and  the  user  of  training  devices  to  evaluate  and  monitor,  the  cost  and  training 
effectiveness  of  training  devices.  The  comprehensive  cost-effectiveness  model  consists  of 
five  submodels  to  perform  media  selection,  assessment  of  effectiveness,  estimation  of 
costs,  cost-effectiveness  trade-offs,  and  device  evaluation.  Each  submodel  is  discussed 
briefly  in  turn  below. 

a.  Media  Selection 

Submodel  1  produces  media  alternatives  for  training  devices  which  first  imdergo  a 
rough  cost-utility  analysis.  It  is  applicable  as  early  as  the  "concq)t  phase"  of  the  weapons 
system  development  cycle,  when  training  concepts  must  be  formulated. 

b.  Assessment  of  Effectiveness 

Submodel  2  consists  of  a  taxonomy  of  functions  and  measures  of  the  effectiveness 
of  training  devices.  The  purpose  of  Submodel  2  is  to  predict  in  detail  the  effectiveness  of 
alternate  training  devices.  It  is  applicable  when  specific  technical,  tactical,  and  logistics 
data  of  the  weapons  system  are  available,  so  that  training  concepts  can  be  formulated  and 
training  devices  can  be  developed. 


c.  Estimation  of  Costs 

Submodel  3  is  a  life-cycle  cost-planning  model  containing  a  cost  breakdown 
structure,  computing  rules,  and  rough  planning  figures. 

d.  Cost-Effectiveness  Trade-offs 

Submodel  4  relates  the  speciHc  effectiveness  of  alternative  training  devices  to  life- 
cycle  cost  predictions.  Its  purpose  is  to  compare  training  effectiveness  with  cost 
development  and  to  identify  cost  drivers  as  well  as  minimally  effective  improvements. 

e.  Device  Evaluation 

Submodel  5  monitors  cost-effectiveness  during  the  in-service  phase  of  the  training 
device  evaluation  and  produces  data  which  can  be  used  as  an  "audit  trail"  for  better 
predictions  and  further  decisions.  It  is  included  in  this  comprehensive  model  to  ensure  that 
early  predictions  and  post  hoc  analyses  use  the  same  conceptual  framework  and 
methodology. 

We  believe  this  model  to  be  the  most  comprehensive  we  have  encountered  in  our 
review  of  the  training  cost-effectiveness  literature.  However,  it  is  much  too  detailed  and 
data-intensive  for  use  by  the  DoD-level  decision-maker  for  training  policy. 

6.  Economic  Issues  in  Cost-Effectiveness  Analysis  of  Military  Skill 
Training  (Solomon,  1986) 

Solomon  (1986)  addressed  the  econmnic  issues  in  cost-effectiveness  analyses  of 
military  skill  training  making  two  important  points  relevant  to  TFDM  development:  1)  if 
there  is  more  than  one  way  to  achieve  an  objective,  q>ecified  as  a  particular  level  of  output 
and/or  effectiveness  (perfacmance),  the  least-cost  alternative  should  be  identified;  and  2)  if 
there  is  a  budgetary  constraint,  there  should  be  a  way  of  selecting  the  alternative  that  will 
result  in  maximum  output  and/or  effectiveness.  The  author  introduces  two  notions  of 
Sassone  (undated),  the  Compensating  Budget  Variation  (CBV)  and  the  Equivalent  Budget 
Variation  (EBV),  each  defined  as  follows; 

•  The  CBV  is  the  maximum  reductitxi  in  the  training  budget  that  will  maintnin  the 
current  level  of  training  effectiveness  if  the  new  type  of  training  is  introduced, 
and 

•  The  EBV  is  the  minimum  increase  needed  in  the  training  budget  to  achieve  the 
same  effectiveness  level  without  tire  new  training. 


Solomon  and  Sassone's  ideas  about  conducting  cost-effectiveness  analyses  are 
consistent  with  one  another  and  with  those  presented  by  the  other  authors  reported  on  here. 

7.  Empirical  Studies  on  Cost-Effectiveness  of  Military  Training  (Fletcher 
and  Orlansky,  1989) 

Fletcher  and  Orlansky  (1989)  summarized  22  recent,  empirical  studies  on  the  cost- 
effectiveness  of  military  training.  Of  the  16  studies  that  compared  simulator  trained 
students  with  students  trained  using  actual  equipment,  four  reported  superior  performance 
for  the  former,  four  for  the  latter,  and  eight  reported  no  difference.  Despite  these  mixed 
results,  several  of  the  studies  indicated  that  simulator  training  costs  less  than  training  using 
actual  equipment  Only  eight  of  the  22  studies  provided  cost  data.  Twelve  studies 
provided  Training  Cost  Ratios  (TCRs).  Of  the  12  TCRs  reported,  11  were  1.0  or  less. 
The  high  TCR  of  2.6  was  based  on  initial  acquisition  of  a  single  trainer.  TCRs  were 
computed  using  life-cycle  costs,  acquisition  costs,  and  hourly  operating  costs.  Also  four 
CERs  were  reported  ranging  from  0.11  to  0.74.  The  authors  were  able  to  conclude  that 
training  using  simulation  was  cost-effective. 

B .  roENTIFICATION  OF  DATA  REQUIREMENTS  AND  DATA  SOURCES 
FOR  COST-EFFECTIVENESS  ANALYSES  OF  MILITARY  TRAINING 
OPTIONS 

In  this  section  we  attempt  to  identify  in  more  detail  those  data  that  are  needed  to 
accomplish  cost-effectiveness  analyses  of  military  training  options  using  the  TFDM.  The 
identification  of  data  requirements  and  sources  is  based  chi  our  review  of  the  recent  training 
cost-effectiveness  literature  and  our  ongoing  experience  in  conducting  the  case  study  to  be 
presented  later  in  this  paper.  The  two  types  of  data  needed  to  feed  the  TFDM  are  training 
cost  and  training  effectiveness  data.  A  complicating  factor  is  that  we  will  need  not  only 
past,  current,  and  projected  near-term  training  cost  and  effectiveness  measures,  but  we  will 
also  need  longer  term  estimates  of  training  cost  and  effectiveness.  These  projections  will 
be  required  for  the  individual  trainee  engaged  in  any  modified  training  regimen  that  might 
result  from  training  funds  investment  using  new,  relatively  unproven  technology. 

1 .  Current  Training  Regimen  Cost  Data  Requirements  and  Sources 

This  section  addresses  data  requirements  and  sources  for  determining  the  cost  and 
effectiveness  of  an  existing  training  regimen.  It  continues  by  defining  the  data 
requirements  and  sources  for  potential  enhancements  to  that  training  regimen  or  some 
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totally  new  training  option.  Such  a  dichotomy  is  necessary  because  of  the  speculative 
nature  of  much  of  the  data  that  will  be  available  on  any  new  training  regimen. 

a.  Data  Requirements 

The  major  categories  of  resources  required  to  support  various  methods  of 
instruction  are  provided  in  Table  III-l  from  Orlansky  and  String  (1979).  As  previously 
noted,  the  authors  were  evaluating  the  cost  and  effectiveness  of  computer-based  instruction 
for  military  training  using  approximately  30  studies  conducted  since  1968.  Unfortunately, 
they  found  that  direct  comparisons  of  the  cost  and  effectiveness  of  different  methods  of 
instruction  were  not  possible  because  of  incomplete  cost  data. 

Table  III-l.  Resources  Required  to  Support  Various  Methods  of  Instruction 

(Major  Categories  Only) 

Resource  (Type  or  Function) 

Program  Development 
Program  Design 
Instructional  Materials 
Conventional 
Individualized  Instruction 
Programming 
First-Unit  Production 
Computer-Based 
Programming 
Coding 

Program  Delivery 
Instruction 

Instructors 

Instructional  Support  Personnel 
Equipment  and  Services 

Laboratory  (Including  Simulators) 

Media  Devices 
Computer  Systems 
Communications 

Materials  (Including  Consum^}les} 

FaciNties 

_ Program  Management  and  Administration _ 

Student  Personnel 

Pay  and  MIowances 

Other  (Temporary  Duty.  Permanent  Chance  of  Station,  etc.) 


Source:  Orlansky  and  String,  1979. 


Knapp  and  Orlansky  (1983)  produced  a  report  identifying,  structuring,  and 
deHning  a  list  of  cost  elements  that  they  believed  fully  described  the  life-cycle  cost  (LCC) 
of  any  formal  program,  course,  or  device  for  individual  training  of  DoD  personnel, 
regardless  of  the  assumptions  imposed  by  the  particular  application.  This  comprehensive 
cost  element  structure  was  developed  to  satisfy  a  widely  recognized  need  for  consistent, 
comparable,  and  credible  dimensions  for  the  evaluation  of  the  cost-effectiveness  of 
alternative  methods  of  training.  Their  cost  element  structure  was  developed  using  a  number 
of  authoritative  and  widely  accepted  cost  guides  issued  by  and  for  the  training  and  weapons 
system  communities  within  the  DoD.  Their  list  also  incorporated  many  recommendations 
of  reviewers  involved  in  related  functions  and  activities. 

The  authors  believed  that  the  general  use  of  a  comprehensive  cost  element  structure 
such  as  theirs  would  offer  several  advantages  for  evaluating  the  costs  of  institutional 
training  programs,  courses,  and  devices.  They  listed  the  following  six  advantages,  of 
which  the  second  is  most  important  here. 

•  A  comprehensive  cost  element  structure  would  ensure  that  all  costs  incurred 
during  the  life  cycle  (or  period  of  interest)  of  a  training  program,  course,  or 
device  would  be  taken  into  account.  Gaps  in  essential  data  could  be  identified 
in  this  way. 

•  It  would  permit  reliable  and  valid  comparisons  among  training  options. 

•  It  would  provide  a  level  of  detail  adequate  to  identify  the  major  cost  elements, 
thus  focusing  attention  on  areas  for  significant  cost  reduction  or  for  trade-off 
analyses  between  high-cost  items  and  training  effectiveness. 

•  It  would  enable  individuals  with  specific  responsibilities  that  interact  with 
training  (e.g.,  budgeters,  manpower  planners,  procurement  specialists)  to 
focus  on  specific  elements  of  primary  interest  to  them  and  to  evaluate  the 
implications  of  those  resources  in  a  total  program  context 

•  It  would  assist  in  identifying  significant  variables  for  use  in  the  development  of 
cost-estimating  relationships  applicable  to  training  programs. 

•  It  would  enhance  communication  and  understanding  among  people  concerned 
with  various  aspects  of  weapon  system  programs  on  subjects  of  mutual 
concern. 

Knapp  and  Orlansky's  Ci)st  Element  Structure  is  attached  as  Appendix  A.  Of 
particular  interest  to  this  effort  is  Element  C,  Operating  and  SuppOTt  Ctosts,  including  direct 
and  indirect  costs.  The  direct  costs  include  instructional  costs,  training  activity  costs, 
airfield  and  carrier  operations  costs  (if  any),  student  costs,  and  other  direct  costs. 


A  number  of  recent  studies  have  tended  to  support  these  assumed  advantages  and 
the  structure  itself.  For  example,  Solomon  (1986)  reported  on  economic  issues  in  the  cost- 
effectiveness  analysis  of  military  skill  training.  Solomon  noted  that  conventional  analyses 
of  training  costs  hinge  on  direct  costs  (personnel,  instructors,  and  materials  and  supplies), 
indirect  costs  (e.g.,  support  personnel,  base  support),  and  investment  costs  (research, 
development,  and  equipment).  Therefore,  when  dollar  values  for  the  categories  listed 
above  are  available,  the  values  can  be  tallied  and,  given  the  number  of  students  per  training 
cycle,  an  average  cost  per  student  can  be  computed.  Solomon  writes;  "When  these  data 
are  available  in  either  time-series  or  cross-sectional  form,  it  may  be  possible  to  derive  'cost¬ 
estimating  relationships,'  employing  some  form  of  multivariate  analysis."  He  cites  Samers 
(1974)  as  a  good  example  of  this  form  of  cost  analysis  as  applied  to  on-the-job  training. 

Using  the  Knapp  and  Orlansky  cost  element  structure,  Fletcher  and  Orlansky 
(1989)  reviewed  recent  studies  of  the  cost-effectiveness  of  military  training.  Their  general 
flndings  with  respect  to  cost-effectiveness  were  reviewed  above.  However,  it  is  important 
to  note  that  they  found  these  cost  elements  were  used  inconsistently  across  the  22  analyses 
considered,  even  though  many  of  the  cost  elements  were  used  in  these  studies. 
Specifically,  the  four  types  of  costs  reported  included  initial  investment  and  acquisition 
costs,  student  time  savings,  operating  costs,  and  life-cycle  costs.  This  finding  again  makes 
the  point  that  cost-effectiveness  figures  in  the  literature  for  similar  training  activities  may 
not  be  comparable  because  the  cost  metrics  were  constructed  from  different  elements. 

These  three  studies  show  that  most  of  the  cost  data  elements  necessary  to  perform 
valid  and  reliable  training  cost-effectiveness  analysis  were  available  6  years  ago.  However, 
based  on  the  recent  study  by  Fletcher  and  Orlansky  (1989),  it  appears  that  much  of  the 
research  community  was  unaware  of  the  utility  of  these  elements  in  performing  appropriate 
and  meaningful  analyses.  Since  then  a  number  of  recent  studies  have  been  conducted  to 
refine  those  cost  data  elements  necessary  for  the  development  and  maintenance  of  a  cost 
data  element  repository  at  the  Training  Performance  Data  Onter  (TPDC),  Orlando,  Rorida. 

Hosek  and  Peterson  (1988)  of  RAND  defined  what  they  referred  to  as  the  initial 
skill  training  data  base  files  including  Military  Occupational  Specialty  (MOS),  initial  skill 
training  number  (ISTN),  and  time  period.  The  four  categories  of  cost  data  accessible  by 
these  identiHers  were  personnel  costs,  equipment  costs,  course  costs,  and  base  operations 
costs.  When  their  report  was  produced  in  November  1988,  further  studies  were  under  way 
and  the  construction  of  such  a  database  at  TPDC  was  yet  to  begin. 
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As  a  part  of  the  IDA  effort,  Gibson  and  Orlansky  (1988)  suggested  that  the  short¬ 
term,  best  option  for  a  data  base  to  support  what  they  refer  to  as  the  cost-production  aspect 
of  training  system  operation  would  be  one  in  which  fairly  simple,  descriptive  data  elements 
were  established  and  organized  in  a  user-oriented  manner.  They  state  that  these  data  bases 
should  be  capable  of  accurately  tracking  and  summarizing  budget  allocations  for  training 
costs,  student  loads,  and  inventory  projections.  Additional  data  bases  should  be  capable  of 
providing  descriptive  summaries  of  training  facilities  and  equipment  including  capabilities, 
capacities,  location,  and  level  of  utilization.  Gibson  and  Orlansky  conclude  that  efforts 
should  be  made  to  list,  summarize,  and  simplify  the  data  in  periodic  and  on-demand  reports 
to  aid  in  the  development  of  training  policy. 

Gibson  et  al.  (1989)  attempted  to  identify  the  types  of  data  and  the  data  base 
structure  needed  to  examine  a  particular  training  policy  question  presented  as  a  case  study. 
They  concluded  that  the  case  study  was  not  fully  addressable  with  the  current  TPDC  data 
bases.  They  go  on  to  state  that  the  need  for  descriptive  information  pertaining  to  courses, 
training  equipment,  and  trained  personnel  is  being  addressed  by  TPDC  data  base 
development  efforts.  Gibson  et  al.  suggested  that  the  data  bases  should  also  contain  a 
detailed  breakdown  of  information  in  the  following  categories: 

•  Student  costs  (time  and  wage  rate) 

•  Instructor  costs 

•  Course  design 

•  Equipment  costs 

•  Facilities  costs 

•  Cost  of  technology,  if  new  R&D  is  required. 

Gibson  et  al.  conclude  that  decision-making  requires  information  concerning  both 
the  long  run  average  and  the  long  run  marginal  cost  to  train  an  individual.  As  of  May 
1989,  data  bases  to  collect  information  on  training  system  utilization  were  being 
conceptualized  and  prototyped  at  TPDC. 

b .  Existing  and  Planned  Sources  of  Training  Cost  Data 

For  the  purpose  of  supporting  the  requirements  of  cost-effectiveness  analyses,  the 
two  most  relevant  TPDC  data  bases  are  the  Macro  Cost  Data  Base  and  the  Training  Cost 
Data  Base.  The  existing  Macro  Cost  Data  Base  is  a  budget-oriented  compilation  of  training 
resources.  Recent  enhancements  of  this  system  include  updater*  -Iscal  Year  Development 


Plan  (FYDP)  data  in  dollars  and  manpower,  a  more  extensive  breakdown  of  the  budget 
figures,  civilian  data  from  within  the  operations  and  maintenance  (O&M)  accounts,  cost 
and  workload  information  from  the  program  element  information  contained  in  the 
Manpower  and  Training  Research  Information  System  (MATRIS),  and  an  automated 
Standard  Reporting  System. 

According  to  Gibson  et  al.  (1989)  the  Macro  Cost  Data  Base  may  be  a  model  for 
some  of  the  other  data  bases.  Its  report-producing  capabilities  have  expanded 
progressively  and  now  it  routinely  produces  valuable  products  for  its  users.  Its  strength  is 
in  its  high-level,  resource  oriented  data  presentation.  Its  most  critical  weakness  for  our 
purposes  is  that  it  does  not  provide  a  cost-effectiveness  justiHcation  for  possible  funding 
changes.  Gibson  et  al.  believe  that  when  this  "top-down"  data  base  can  be  coordinated 
with  several  of  the  planned  "bottom-up"  data  bases,  a  wide  range  of  training  cost- 
effectiveness  policy  issues  will  be  directly  addressable. 

A  well-organized  and  reasonably  complete  data  structure  has  been  developed  for  the 
Macro  Cost  Data  Base,  and  the  sources  of  many  of  the  data  elements  have  been  identified. 
When  populated,  this  structure  will  allow  calculation  of  the  average  and  marginal  cost  of 
individual  course  graduates  and  total  cost  of  alternative  course  throughputs  for  recruit, 
initial,  and  advanced  skill  training  for  both  individuals  and  units.  Also,  recruiting  bonuses, 
special  pay,  and  separation  cost  data  would  be  collected.  This  would  allow  individual 
replacement  costs  to  be  calculated  more  accurately. 

c.  Summary 

With  respect  to  requirements  and  sources  for  current  training  cost  data,  it  appears 
that  the  required  categories  are  well  understood  and  that  the  specific  requirements  for  the 
necessary  training-cost  data  bases  to  make  these  data  accessible  are  rapidly  taking  shape. 
Unfortunately,  only  limited  portions  of  the  required  data  bases  have  been  implemented  and 
plans  for  moving  forward  with  this  critical  need  are  relatively  vague.  For  example,  the 
recent  draft  TPDC  Information  Architecture  Overview  (TPDC,  1989)  does  not  discuss  cost 
data  to  the  degree  and  depth  of  detail  that  we  believe  will  be  necessary. 


2.  Current  Training  Regimen  Performance  Data  Requirements 
and  Sources 

a.  Requirements 

Gibson  and  Orlansky  (1986)  reviewed  and  summarized  17  studies  concerned  with 
the  collection  of  objective  performance  data  far  the  purpose  of  evaluating  the  effectiveness 
of  maintenance  training.  They  discussed  their  results  in  relation  to  rive  major  aspects  of 
evaluating  the  effectiveness  of  training:  1)  transfer  of  training;  2)  quality  of  simulation;  3) 
effects  of  training  on  individual  performance;  4)  differential  effects  of  alternative  methods 
of  training;  and  5)  the  effects  of  training  and  experience  on  unit  performance  and 
operational  readiness.  It  is  important  to  note  Gibson  and  Orlansky's  conclusion  that  most 
of  the  training  effectiveness  literature  has  only  marginal  usefulness  in  formulating  training 
policy  since  it  tends  to  focus  on  the  early  effects  of  training.  Consequently,  increased  data 
collection  efforts  should  be  directed  toward  assessing  the  long-term  effects  of  training  and 
experience  on  the  quality  of  individual  performance,  unit  effectiveness,  and,  ultimately,  on 
combat  readiness. 

Most  of  the  17  studies  and  measiues  reviewed  by  Gibson  and  Orlansky  (1986) 
were  designed  to  assess  the  effects  of  formal  training  and  experience  on  maintenance 
performance.  Because  of  the  costs  in  time  and  eriort  needed  to  produce  skilled  technicians, 
it  was  not  only  reasonable  but  essential  to  consider  whether  the  training  methods  and 
devices  had  any  real  effect  on  maintenance  performance.  Only  eight  of  the  studies 
presented  used  maintenance  performance  measures  to  determine  whether  training  made  any 
measurable  difference  in  productivity.  Three  of  the  training  effectiveness  measurement 
studies  compared  the  performance  of  maintenance  technicians  who  were  trained  on 
maintenance  training  simulators  to  the  performance  of  those  trained  on  the  actual 
equipment 

Solomon  (1986)  observed  that  there  are  significant  problems  in  the  representation 
and  measurement  of  training  effectiveness.  One  approach  is  to  follow  current  practice  and 
use  a  level  of  knowledge  and  skill  represented  by  a  score  resulting  from  one  or  more  tests 
administered  during  the  course.  Solomon  concluded  that  information  on  effectiveness  is 
limited  generally  to  student  performance  at  school  despite  the  fact  that  the  more  important 
and  revealing  measurement  of  training  effectiveness  is  performance  of  the  individual  on  the 
job.  Unfortunately,  as  Solomon  noted,  it  has  rarely  been  possible  to  relate  empirically  the 
level  of  achievement  in  school  to  job  performance  (productivity).  Hence,  he  concluded  that 
it  has  not  been  possible  to  relate  alternative  training  methods  in  schools  to  job  perframance. 
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According  to  Gibson  and  Orlanslcy  (1988),  a  data  base  that  supports  policy-making 
efforts  on  the  basis  of  performance  effectiveness  data  should  provide  information  on  the 
skill  levels  achieved  through  initial  skill  training,  on-the-job  training,  and  post-initial  skills 
individual,  team,  and  unit  training.  Gibson  and  Orlansky  further  observe  that  the 
performance  effectiveness  data  requirements  can  be  divided  into  two  categories: 

1.  Data  that  relate  the  performance  levels  achieved  through  formal  training 
programs  to  the  skill  levels  and  mixes  required  by  the  operating  forces,  and 

2.  Data  that  relate  post-initial  skills  training  activities  (such  as  flying  hours)  to 
maintaining  or  improving  skill  levels  while  performing  in  the  operational 
environment. 

Test  scores  and  course  grades  satisfy  part  of  the  first  requirement  and  they  provide 
a  basis  for  judging  the  effects  of  changes  to  the  training  program  provided  that  the  same 
tests  are  used  to  measure  the  training  results.  In  our  case,  the  test  data  would  be  much 
more  valuable  if  they  could  be  directly  equated  to  definable  levels  of  apprentice  or 
journeyman  performance  in  the  fleld.  If  equivalencies  could  be  established,  it  would 
become  possible  to  definitively  link  the  costs  of  training  to  speciHc  levels  of  competence, 
which  are  what  we  need  for  our  analysis. 

In  their  review  of  22  recent  studies  on  the  cost-effectiveness  of  military  training, 
Fletcher  and  Orlansky  (1989)  found  that  the  types  of  cost  and  effectiveness  data  collected  in 
these  studies  did  not  permit  examination  of  trade-offs  between  cost  and  effectiveness.  The 
range  of  effectiveness  measures  considered  by  each  individual  study  was  narrow.  None  of 
the  studies  considered  all  five  of  the  categories  listed  by  (Thatelier  et  al.  (1985).  Moreover, 
only  seven  studies  considered  attitudes  of  either  students  or  instructors;  only  three 
considered  student  attrition;  and  none  considered  on-the-job  performance.  On  the  other 
hand,  all  of  the  studies  considered  end-of-course  effectiveness,  and  all  but  two  of  the 
studies  considered  transfer  of  training  from  simulates  to  real  equipment  Unfortunately, 
student  time  savings  through  use  of  actual  equipment  was  treated  as  both  a  cost  and  an 
effectiveness  measure.  Eight  of  the  22  studies  measured  time  to  complete  a  task  using 
actual  equipment  and  four  other  studies  measured  success  in  maintaining  ot  operating  actual 
equipment  as  an  indicator  of  effectiveness. 

b.  Available  Sources  of  Training  Effectiveness  Data 

Do  the  necessary  data  and/or  data  bases  exist  to  facilitate  a  systematic  atten^t  at  the 
measurement  of  training  effectiveness  (i.e.,  individual  productivity/ptofrciency  change  that 
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results  from  training)?  String  and  Orlansky  (1979)  concluded  that  then-current  large-scale 
maintenance  information  systems  did  not  provide  the  required  information  about  individual 
performance  needed  to  relate  individual  training  to  productivity.  Gibson  and  Orlansky 
(1988)  and  Gibson  et  al.  (1989)  were  still  concluding  that  little  had  been  done  to  assemble 
the  necessary  data  bases  to  address  the  training  effectiveness  measurement  problem. 
Gibson  et  al.  did  find  that  the  TPDC  data  bases  contain  some  performance  data  on 
personnel  characteristics  which  may  predict  the  quality  of  achievement  in  school  or  in  the 
Held  and  some  data  on  the  effectiveness  of  the  various  training  systems  in  satisfying 
approved  training  requirements  or  on  the  number  of  people  completing  established  training 
syllabi.  However,  they  found  little  or  no  performance  data  addressing  the  effects  of 
training  courses  or  techniques  on  the  quality  of  individual  or  group  achievement  either  at 
the  end  of  the  course  or  under  operational  conditions.  It  is  these  latter  data  that  are  most 
critical  for  our  proposed  analysis.  The  authors  go  on  to  suggest  short-,  intermediate-,  and 
long-term  strategies  for  developing  the  necessary  performance  measures  and  supporting 
data  bases. 

Gibson  and  Orlansky  (1988)  believe  that  the  most  important  long-term  strategy 
must  be  to  develop  performance  measures  that  relate  training  achievement  levels  to 
operational  apprentice  or  journeyman  skill  levels.  Only  if  this  can  be  achieved  will  it  be 
possible  to  develop  meaningful  cost-effectiveness  trade-offs  for  training.  Gibson  and 
Orlansky  suggest  that  the  commands  that  receive  the  new  trainees  could  be  tasked  to 
provide  feedback  on  training  effectiveness  with  these  students  by  routinely  producing  Net 
Productivity  Ratings  of  first-term  personnel  assigned  to  the  command.  These  ratings 
would  partially  fulfill  the  above  objective.  However,  they  require  a  minimum  delay  of  18 
months  to  2  years  to  assess  the  effects  of  training  programs  on  the  quality  of  performance, 
and  they  confound  student  and  supervisor  productivity  information,  partially  masking  the 
specific  effects  of  training  on  student  peifonnance. 

In  agreement  with  the  present  authors,  Gibson  and  Orlansky  (1988)  believe  that  a 
scale  relating  training  achievement  to  operational  skill  levels  would  provide  immediate 
feedback.  It  could  also  be  used  to  determine  the  costs  of  training  to  a  specified  skill  level. 
This  would  make  many  training  decisions  more  directly  analogous  to  hardware  cost- 
effectivenet ..  trade-offs  in  which  the  costs  of  training  to  a  specified  criterion  could  be 
weighed  against  the  anticipated  operational  benefits.  Finally,  they  note  that  the  long-term 
strategy  should  also  encourage  the  development  and  validation  of  highly  aggregated 
measures  of  the  effects  of  collective  training  (e.g.,  training  as  a  force  multiplier). 
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c.  Summary 

In  summary,  with  respect  to  the  data  requirements  and  data  sources  necessary  to 
measure  current  training  regimen  effectiveness,  despite  a  growing  appreciation  of  the 
requirements,  very  litde  has  been  done  during  the  past  decade  to  create  the  necessary  on¬ 
line  data  bases  to  support  them.  In  particular,  the  current  draft  TPDC  Information 
Architecture  Overview  (TPDC,  1989)  contains  very  little  of  the  necessary  data. 

3 .  Cost  Data  Requirements  and  Sources  for  Modified,  Enhanced,  or  New 
Training  Regimens 

The  previous  two  subsections  addressed  the  measurement  of  costs  and  training 
effectivenesses  of  current  training  regimens,  i.e.,  those  for  which  some  historical  data 
should  exist.  To  make  funding  decisions  about  contemplated,  new  training  regimens 
requires  a  priori  estimates  of  their  cost  and  effectiveness.  This  subsection  and  the  next 
address  the  special  issues  involved  in  such  a  priori,  comparative  analyses. 

a.  Requirements 

The  key  to  measuring  or  estimating  the  cost  of  modified  training  regimens  is  to 
accurately  determine  the  cost  of  the  new  technology  to  be  inserted  in  addition  to  or  in  place 
of  the  current  technology  and  to  estimate  the  effects  of  such  insertion  on  all  other  training 
costs.  In  so  doing,  we  face  essentially  the  same  difficulties  we  faced  when  attempting  to 
measure  the  cost  of  any  current  trairung  regimen,  compounded  by  the  uncertainties  of  the 
modified  trairung  regimen's  effect  on  time  to  train.  Up  to  now,  most  cost  savings  are  due 
to  reductions  in  time  needed  to  train  persormel  to  a  specified  level  of  proficiency.  Any  cost 
analysis  we  perform  for  a  modified  trairung  regimen  must  be  focused  on  the  particular 
new/existing  technology  in  which  we  are  planning  to  invest,  for  example,  CBI/IVD. 
Typically,  we  have  been  required  to  look  for  data  in  the  research  literature. 

In  the  previously  referenced  report  on  recent  studies  of  the  cost-effectiveness  of 
military  training  in  TTCP  countries,  Retcher  and  Orlansky  (1989)  found  that  even 
researchers  attempting  to  measure  the  relative  cost-effectiveness  of  new  approaches  to 
training  fail  to  use  an  adequate  set  of  cost  data.  Only  4  of  10  possible  cost  categories 
(Table  111-2)  were  considered  in  their  set  of  studies.  They  ccmcluded  that  the  cost  data  were 
incomplete  (and  potentially  misleading).  Only  8  of  the  22  studies  purporting  to  measure 
cost-effectiveness  provided  any  cost  data  at  all! 


Table  III-2.  Effectiveness  and  Cost  Data  Provided  by  22  Empirical  TTCP  Studies 


Types  of  Data 

Number  of  Studies 

Effectiveness  Measures 

End  of  Course 

22 

Transfer 

20On-the-Job 

0 

Student  Attrition 

3 

Instructor  Attitudes 

7 

Student  Attitudes 

7 

Cost  Data 

Research  and  Development 

0 

Initial  Investment 

Program  Development 

0 

Acquisition 

3 

Operating  and  Support 

Student  Pay  and  Allowance 

0 

Instructor  Pay  and  Allowance 

0 

Maintenance  and  Repair 

0 

Program  Modification 

0 

Student  Time  Savings 

8 

Operating 

4 

Life-Cycle  Costs 

(includes  all  of  the  above) 

2 

Source:  Fletcher  and  Orlansky,  1989. 

b.  Sources 

There  are  few  sources  from  which  to  derive  the  r;ost  data  required  for  predicting  the 
cost-effectiveness  of  training  regimens  based  on  new  technologies.  An  exception  is  the 
planned  TPDC  CBI/IVD  training  technology  data  base  (TTDB).  Cost  models  and  cost¬ 
estimating  procedures  will  be  required  to  fulfill  our  needs.  We  can  also  extrapolate  from 
historical  data  for  similar  technologies. 

c.  Summary 

In  summary,  with  respect  to  the  modified  training-cost  data  requirements  and 
sources,  we  face  a  more  difficult  situation  than  we  did  for  existing  training  regimens.  The 


categories  of  cost  data  required  will  emain  basically  unchanged  except  for  a  greater 
emphasis  on  up-front  research,  development,  and  acquisition  costs. 

4.  Performance  Data  Requirements  and  Sources  for  Modified  Training 
Regimens 

Cost  data  requirements  and  sources  for  modified  training  regimens  are  intimately 
related  to  performance  data  requirements  and  sources.  This  is  because  any  cost  savings 
from  the  new  technology  is  most  likely  to  come  from  the  increased  efficiency  of  the 
new/modified  training  regimen  in  providing  the  necessary  training.  Efficiency  in  providing 
training  is  direcdy  related  to  the  performance  proficiency  that  results  from  training.  The 
more  rapidly  the  latter  rise  to  criterion  for  the  trainee,  the  more  efficient  we  say  the  subject 
training  regimen  is.  By  necessity,  our  primary  focus  here  must  be  on  techniques  and 
methods  for  measuring  the  differential  training  effectiveness  of  alternative  training  regimens 
as  indicated  in  the  literature  reviewed  below. 

a.  Data  Requirements 

We  now  present  the  findings  from  a  number  of  studies  that  focus  on  training 
performance  data  requirements,  particularly  with  respect  to  measuring  the  effects  of 
modified,  enhanced,  or  entirely  new  training  regimens,  all  typically  making  use  of 
relatively  new  technologies.  The  emphasis  is  on  the  measurement  of  the  differential  effects 
of  alternative  training  options,  that  is,  comparative  studies. 

Orlansky  and  String  (1979)  suggest  measurement  of  performance  of  graduates  on 
jobs  in  the  field  as  a  sound  approach  to  postgraduate  monitoring  of  students  after  their 
assignment  to  duty  stations.  Such  data  are  not  now  collected  systematically.  In  their 
review  of  30  studies  of  CAI/CMI,  the  authors  found  the  following  measures  of 
effectiveness  used  Iot  comparative  puiposes: 

1 .  Student  achievement  on  tests  administered  during  and/or  at  the  end  of  course 

2.  The  length  of  time  required  for  students  to  conq)lete  a  course 

3 .  Academic  attrition  rates 

4 .  Student  and  instructor  attitudes. 

One  measure  of  the  differential  effectiveness  of  training  regimens  that  might  be 
more  seriously  considered  relates  to  observed  differences  in  training  time  periods  for 
different  training  techniques  and  methodologies,  for  example,  the  use  of  simulators  versus 
actual  equipment.  In  a  number  of  cases,  it  has  bwn  possible  to  estimate  these  differences 
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(see  Orlansky  and  String,  1979, 1981).  The  limitation  has  been  the  difficulty  in  obtaining 
information  on  how  these  measures  relate  to  on-the-job  performance.  Nevertheless, 
observed  differences  in  the  duration  of  required  training  periods  represent,  or  at  least 
suggest,  differences  in  cost. 

Gibson  and  Orlansky  (1986)  also  addressed  the  measurement  of  the  differential 
effects  of  training  methods.  They  focused  on  the  comparative  differences  between 
simulators  and  actual  equipment  for  training,  which  has  been  a  major  source  of  concern  and 
controversy  in  the  maintenance  training  community.  From  a  total  of  17  studies,  they 
examined  closely  three  concerning  differential  training  effectiveness  measurement.  The 
measures  used  in  these  studies  included  troubleshooting  interview,  behaviorally  anchored 
rating  scales,  and  speed  of  diagnosing  malfunctions.  Gibson  and  Orlansky  note  the  hazards 
of  giving  absolute  interpretations  to  the  ordinal  data  produced  by  the  first  two  measures. 
Gibson  and  Orlansky  also  reviewed  a  number  of  job  performance  measures  that  could  be 
used  to  evaluate  relative  maintenance  training  effectiveness.  Among  those  identiHed  were 
speed  of  completing  maintenance  work  and  net  productivity  ratings.  Both  metrics  were 
sensitive  to  differences  in  training  techniques  [see  Johnson,  et  al.  (1983),  McConnell  and 
Johnson  (1984),  and  (fester  and  Marcus  (1985)]. 

Solomon  (1986)  notes  that  an  alternative  to  using  direct  on-the-job  measurements  of 
productivity  to  measure  the  relative  effectiveness  of  training  methods  is  to  use  relative 
standards  of  achievement.  Unfortunately,  this  is  not  a  viable  measure  for  technology  that 
has  not  yet  been  inserted.  However,  if  we  believe  that  we  have  enough  existing, 
comparable  cases  of  insertion  of  the  same  technology  or  a  highly  representative  one,  then 
we  might  wish  to  consider  the  use  of  relative  standards  of  achievement  to  measure  the 
differential  proficiency  of  alternative  training  regimens. 

Solomon  reviewed  the  study  by  (Juester  and  Marcus  (1985),  which  used  survey 
data  and  provides  a  good  example  of  the  application  of  a  relative  standard.  By  using  data 
from  the  Enlisted  Utilization  Survey  and  Navy  administration  records,  they  were  able  to 
compare  the  cost-effectiveness  of  formal  school  instruction  and  on-the-job  training.  The 
measure  of  effectiveness  was  net  productivity  of  the  individual  at  different  times  at  the  duty 
station.  ("Net"  refers  to  subtracting  the  loss  in  production  of  more  experienced  supervisory 
personnel  required  to  train  these  individuals.)  This  assessment  by  the  supervisors  was 
made  relative  to  the  average  specialist  with  4  years  of  experience;  it  is  an  example  of  a 
relative  standard.  The  results  of  the  study  suggested  that  school  training  is  more  effective 
and  also  more  cost-effective  than  on-the-job  training. 


Hosek  and  Peterson  (1988)  of  RAND  note  the  importance  of  longitudinal  data  as  a 
key  requisite  for  a  training  effectiveness  data  base.  They  assert  that  longitudinal  data  will 
enable  comparisons  of  cohorts,  evaluation  of  policy  changes,  studies  of  changes  in  skill 
requirements  over  time,  and,  most  importantly  for  our  purposes,  comparison  of  different 
methods  used  over  time  in  the  same  course.  Such  data  are  not  currently  being  maintained, 
although  some  of  it  could  be  reconstructed  from  existing  data  bases. 

The  above  cited  review  by  Fletcher  and  Orlansky  (1989)  concerning  recent  studies 
on  the  cost-effectiveness  of  military  training  in  TTCP  countries  focused  primarily  on 
studies  that  compared  the  effectiveness  and  cost  of  relatively  inexpensive  part-task 
simulators  versus  fiill-task,  highly  realistic  simulators.  As  noted  in  Section  2  (above),  the 
authors  found  the  range  of  measures  considered  by  these  studies  to  include; 

1 .  Student  and  instructor  attitudes 

2.  Student  attrition 

3 .  On-the-job  performance 

4.  End-of-course  effectiveness 

5 .  Transfer  of  training  to  real  equipment 

6.  Time  savings  in  use  of  actual  equipment 

7 .  Success  in  maintaining  or  operating  actual  equipment. 

Gibson  et  al.  (1989)  discussed  the  key  issue  of  qualitatively  different  training 
integral  to  the  new  technology.  They  note  that  new  training  technology  is  often 
implemented  not  only  because  it  allows  the  same  things  to  be  done  for  less  money,  but  also 
because  it  has  capabilities  or  allows  training  strategies  that  are  qualitatively  different  from 
those  previously  available.  A  possible  example  of  this  might  be  that  interactive  videodisc 
(IVD)-delivered  training  could  allow  students  to  perform  actions  that  would  be  harmful  to 
themselves  and/or  the  equipment  on  which  they  would  otherwise  be  training.  While  these 
qualitatively  different  capabilities  may  be  the  key  reasons  for  implementing  such 
technology,  they  make  analyzing  the  cost-effectiveness  of  the  technology  insertion  more 
difficult.  Performance  measurement  will  ultimately  be  necessary  to  quantify  this 
improvement  in  order  to  complete  our  analysis. 


b.  Data  Sources 

As  with  most  of  the  other  categories  of  data  identified  previously,  few  if  any  of  the 
data  are  being  routinely  collected  and  stored  that  are  necessary  for  measurement  of  the 
relative  effectiveness  of  different  training  regimens. 

c.  Summary 

A  number  of  authors  have  begun  to  identify  the  data  necessary  for  comparing  the 
effectiveness  of  different  training  regimens.  However,  few  of  these  data  are  now  being 
systematically  maintained  in  data  bases  or  even  appear  to  be  used  systematically  in 
published  research  studies. 

C .  A  TRAINING  FUNDS  DECISION  METHODOLOGY  (TFDM) 

In  this  subsection  we  make  explicit  the  Training  Funds  Decision  Methodology 
(TFDM)  which  in  Chapter  FV  we  will  apply  to  two  versions  of  a  case  smdy  involving 
electronics  maintenance  training.  This  methodology  draws  upon  a  history  of  ^proaches  to 
cost-effectiveness  analysis  for  military  training.  It  also  takes  into  account  training  cost  and 
effectiveness  data  requirements  and  the  current  realities  of  data  availability  for  cost- 
effectiveness  analysis  of  military  training  options.  In  the  subsections  that  follow,  we  will 
develop  the  five-step  TFDM  procedure.  We  will  then  proceed  with  separate,  more  detailed 
discussions  of  each  step. 

The  basic  five-step  procedure  for  the  TFDM  appears  in  Figure  III-l  and  is  briefly 
described  as  follows: 

1 .  Frame  the  Question-based  on  the  general  concerns  being  raised  in  the  specific 
training-funds  decision  context,  match  the  current  analysis  situation  with  one 
template  or  schema  chosen  from  a  predefined  taxonomy  of  possible  analysis 
situations. 

2.  Make  the  Options  Explicit-based  on  specific  questions  and  concerns, 
including  training  or  performance  objectives,  develop  alternative  training  plans; 
this  may  require  some  up-fiont  analysis  to  ensure  adherence  to  cost  and 
performance  constraints. 

3 .  Determine  Costs  of  Options-determine  the  explicit  anticipated  costs  of  each 
training  option  in  all  relevant  cost  categories;  this  may  require  the  use  of  a  cost 
model  to  provide  a  fair  comparison  of  the  costs  of  the  various  alternatives. 


STEP1 


STEP  2 


STEPS 


Figure  lli-l.  The  Training-Funds  Decision  Methodology  • 

4 .  Determine  the  Training  Effectiveness  of  Each  Option—predict  the  gains  in  job 
performance  productivity/proficiency  that  would  result  from  each  training 
option;  in  some  cases,  the  research  literature  may  be  our  best  and  only  source 

for  such  data.  ® 

5 .  Perform  Cost-Effectiveness  Calculations  and  Draw  Conclusions-determine  the 
productivity/proticiency  gains  to  be  expected  per  unit  cost  for  each  training  option  and  draw 
final  conclusions  regarding  the  most  desirable  option,  taking  appropriate  constraints  (e.g., 

budgetary,  time  to  train)  into  account  • 

We  will  now  proceed  with  a  discussitm  of  each  step  in  the  TFDM. 


1 .  Step  1:  Frame  the  Question 

The  training  funds  decision-maker  will  tning  with  him  to  any  decision  context  or 
policy  analysis  situation  general  concerns  regarding  the  issue(s)  in  question.  At  this  stage 
in  the  TFDM,  the  first  goal  will  be  to  define  and  limit  the  question.  One  way  of 
accomplishing  this  is  to  search  within  a  taxonomy  of  possible  decision  contexts,  for  an 
a  priori  decision-making/policy  analysis  template  (frame  or  schema)  to  match  the  current 
situation.  Once  this  match  has  taken  place,  the  various  components  of  the  selected  analysis 
strategy  can  be  made  explicit,  that  is,  the  frame  can  be  applied.  For  example,  there  might 
be  a  generic  analysis  template  for  decisions  concerning  the  funding  of  increased  R&D  and 
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for  applications  of  a  relatively  new  method  of  instruction  (e.g.,  the  use  of  CBI/IVD 
technology). 

Gibson  et  al.  (1988)  suggested  one  possible  taxonomy  of  training  policy  issues 
having  seven  components.  The  components  and  their  descriptions  are  as  follows: 

•  Training  Type.  Is  the  training  designed  to  develop  individual  skills  or 
collective  skills?  Typically,  analysis  of  individual  training  will  require 
substantially  different  kinds  of  data  than  analysis  of  collective  training. 

•  Performance  Standards.  Does  the  policy  issue  involve  a  change  in  the 
performance  levels  to  be  achieved  through  training?  If  it  does,  data  on 
performance  or  effectiveness  will  certainly  be  required  for  analysis  of  the 
issue.  (Analysis  of  many  policy  issues  that  do  not  involve  changing 
perfcnmance  standards  also  requires  effectiveness  data.) 

•  Training  Time  or  Frequency.  Does  the  issue  involve  how  long  the  training 
should  continue  or  how  frequently  it  should  be  repeated?  Measurement  of  the 
costs  of  the  alternatives  involved  in  issues  of  this  kind  may  be  relatively 
simple,  since  they  involve  just  doing  more  of  something  that  is  already  being 
done.  However,  effectiveness  information  is  necessary  to  analyze  such  issues. 

•  Training  Volume.  Does  the  issue  concern  how  many  people  or  units  should  be 
trained  per  time  period?  Issues  that  involve  only  training  volume  should  be 
particularly  easy  to  analyze,  since  the  effectiveness  data  needed  to  study  them 
are  likely  to  be  easier  to  get  than  data  on  the  performance  implications  of  new 
or  intensified  training. 

•  Training  Technology.  Does  the  issue  revolve  around  a  possible  choice  of 
innovative  training  technologies?  Such  issues  may  require  measurement  of 
research  and  development  costs  and  measurement  of  effectiveness.  They  seem 
certain  to  involve  some  procurement  costs  fen*  instroctional  equipment. 

•  Course  Design.  Does  the  policy  issue  encompass  a  choice  among  alternative 
course  designs?  Even  if  new  technologies  are  not  involved,  data  on  R&D 
costs  are  likely  to  be  necessary  to  analyze  such  issues. 

•  Location.  Is  it  contemplated  that  some  training  will  take  place  at  a  new 
location?  If  so,  information  on  construction  costs  will  probably  be  required  for 
analysis  (as  well  as  information  on  the  costs  of  transporting  individuals  and 
units). 

As  an  example  of  the  approach,  the  previously  cited  Canadian  TMM  implicitly 
assumed  a  partial  taxonomy  of  decision  contexts  involving  at  a  minimum  1 1  components. 
See  Figure  n-5,  above. 


It  is  not  our  purpose  to  develop  the  necessary,  all-inclusive  taxonomy  of  training 
policy  issues  and/or  decision  contexts.  The  point  to  be  made  is  that  such  a  ta.xonomy  will 
allow  us  better  to  anticipate  and  plan  for  the  complexities  of  any  one  analysis,  particularly 
with  respect  to  identifying  the  types  and  sources  of  the  necessary  training  cost  and 
effectiveness  data.  Appendix  B  is  a  summary  of  those  steps  to  be  taken  and  associated 
questions  to  be  asked  in  making  training-policy  decisions,  and  Appendix  C  gives  a  list  of 
training-policy  decision  information  items  for  each  of  the  five  major  steps  in  the  TFDM. 

2 .  Step  2:  Make  the  Options  Explicit 

Once  the  policy  issue  or  training  funds  decision  has  been  fi-amed,  we  move  to 
making  the  options  explicit.  Doing  so  will  typically  require  that  detailed  alternative  training 
plans  be  developed.  These  plans  are  all  comparable  in  that  they  will  be  derived  to  meet 
performance  objectives  which  are  themselves  derived  from  an  analysis  of  the  requirements 
of  the  job  to  be  trained  and  the  tasks  for  which  the  training  is  to  be  conducted.  The  plans 
may  vary  with  respect  to  such  factors  as: 

1 .  The  method  of  delivery  (e.g.,  self-paced  versus  group) 

2 .  The  duration  of  traiiting 

3.  Location  of  training  (c.g.,  school  versus  on-the-job) 

4.  The  type  of  equipment  used  (e.g.,  simulation  versus  actual  equipment). 

Training  options/altematives  must  be  clearly  delineated  so  that  an  explicit  context  is 
provided  for  the  determination  of  both  cost  and  effectiveness.  It  may  be  necessary  to  start 
with  a  baseline  training  option  (usually  the  current  nxxle),  even  though  the  policies/options 
under  consideration  are  all  deviations  from  that  baseline.  The  explicit  time-phasing  of  the 
introduction  of  alternatives  into  the  training  process  will  be  critical  in  wder  to  subsequently 
determine  the  life-cycle  costs.  (Sec  Appendices  B  and  C  for  a  summary  of  the  activities 
involved  in  this  step  and  the  associated  information  items.) 

3 .  Step  3:  Determine  the  Costs  of  Options 

The  average  cost  to  provide  each  training  option  for  an  individual  trainee  must  be 
determined.  Let  us  refer  to  this  quantity  as  the  training  cost  for  option  i,  TCi.  There  are 
obviously  several  different  MOSs  at  several  different  ranks  which  could  be  considered  for 
training  in  a  given  situation.  Let  us  take  one  particular  set  of  skills  taught  to  a  specific  rank 
or  set  of  ranks,  for  example,  electronic  equipment  maintenance  and  repair.  We  need  to 
determine  the  average  cost  to  implement  each  training  option  for  these  skillsfranks  for  one 
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soldier.  In  order  to  do  so,  we  must  first  identify  the  course  or  sequence  of  courses  in 
which  these  ranks  will  acquire  the  particular  skills  and,  through  analysis  and/or  modeling, 
attempt  to  answer  the  following  question;  "What  is  the  cost  to  train  one  individual  (the  cost 
per  graduate)  using  the  training  option  under  consideration,  assuming  simple  formal 
training  costs  and  taking  student  attrition  into  account?" 

Ideally,  this  question  would  be  answered  by  accessing  the  data  base  in  which  the 
necessary  data  are  stored  to  calculate  the  long  run  average  cost  to  train  an  individual 
through  the  particular  training  option.  These  data  should  be  broken  down  into  at  least  the 
following  categories; 

1 .  Student  costs  (time  to  train  and  pay  and  allowance  rate) 

2.  Travel,  food,  ana  lodging  costs 

3.  Instmctor  costs 

4 .  Equipment  and  facilities  costs 

5 .  Other  related  costs,  as  previously  discussed. 

Basically,  we  are  asking  the  question,  "How  much  does  it  cost  to  get  one  person 
through  the  particular  pan  of  the  training  pipeline  on  which  we  are  focusing?"  It  may  be 
necessary  to  aggregate  across  several  courses  in  a  sequence  and  to  take  course  attrition  into 
account.  Nevertheless,  we  are  focusing  on  the  simple  formal  training  costs  and,  if 
necessary,  the  R&D  investments  required  to  bring  training  technology  to  the  level  at  which 
it  will  be  applied  in  these  courses.  We  will  return  later,  in  our  case  study,  to  an  explicit 
consideration  of  the  exact  format  of  cost  information  and  the  sources  for  the  cost  data  that 
we  will  need. 

For  some  options,  it  may  be  necessary  to  determine  the  cost  per  student  for  training 
that  involves  some  new  technology.  In  some  instances  it  may  be  the  case  that  we  will  be 
paying  no  new  R&D  costs  for  this  technology,  that  is,  we  may  be  applying  an  existing 
technology,  but  not  paying  to  develop  new  technology.  In  such  situations,  we  will  be 
purchasing  off-the-shelf  hardware  and  software  and  paying  for  the  development  of 
courseware.  Nonstandard/specialized  hardware  or  software  may  be  necessary  for  our 
application,  and  some  accounting  should  be  made  for  this  possibility. 

If  the  above  situation  holds  true,  we  will  need  to  calculate  how  the  particular 
technology  that  we  will  bring  to  bear  in  this  training  option  will  affect  the  average  cost  to 
run  an  individual  through  the  relevant  portion  of  the  timning  pipeline.  Since  the  necessary 


data  are  not  likely  to  exist,  we  may  need  to  estimate  them  on  the  basis  of  historical  data  that 
reflect  the  costs  in  similar  training  domains,  where  more  or  new  technology  has  been 
brought  to  bear.  We  know,  for  example,  that  equipment  costs  (including  hardware, 
software,  and  courseware)  will  go  up,  but  these  will  be  spread  across  many  individuals  and 
amortized  over  time.  It  is  possible  that  time  to  train  and  associated  costs  will  go  down  if 
the  new  training  regimen  is  more  effective  and  training  occurs  to  a  specific  criterion  level  of 
perfonnance.  It  is  also  possible  that  the  time  to  train  will  be  deliberately  held  constant  and 
the  result  of  the  new  or  added  technology  will  be  seen  in  a  greater  proficiency  incremenL  It 
may  be  that  fewer  instructors  will  be  necessary,  but  possibly  more  facilities  (floor  space) 
for  training  equipment  may  be  required.  All  of  the  potential  trade-offs  must  be  identified 
and  estimates  of  the  cost  per  student  completing  the  training  option  must  be  calculated. 

It  is  possible  that  the  training  cost  for  new  high-technology  training  options  will  not 
be  less  than  the  cost  for  old,  lower  technology  options,  despite  any  savings  in  time  to  train. 
There  is  a  certain  investment  in  the  existing  infrastructure  that  makes  it  very  slow  to 
change.  For  example,  it  is  likely  that  the  duration  of  a  sequence  of  courses  would  be 
modified  only  after  a  great  deal  of  data  is  gathered.  We  may  therefore  find  ourselves 
asking  whether  an  increment  in  proficiency  associated  with  a  training  option,  which 
includes  new  technology,  is  worth  the  increase  in  cost  that  we  will  incur. 

The  above  arguments  notwithstanding,  it  is  possible  that,  given  the  current 
emphasis  on  training  to  criteria,  the  cost  per  person  to  train  might  decrease  under  any 
training  option  involving  new  technologies.  Additionally,  there  are  certain  intangibles  that 
come  along  with  new  technologies,  IVD  for  example,  that  provide  difficult-to-measure 
increases  in  readiness.  Examples  of  such  increases  include  a  greater  ability  to  respond  to 
real-world  situations  due  to  greater  realism  in  the  training.  The  two  versions  of  the  case 
study  to  be  presented  below  will  provide  more  insight  into  these  issues.  Table  111-3 
summarizes  the  major  issues  addressed  above  with  respect  to  the  computation  of  TCi.  (See 
Appendices  B  and  C  for  a  summary  of  the  activities  involved  in  this  step  and  the  associated 
information  items.) 


Table  111-3.  Determine  the  Training  Costs  for  Each  Option  (TCI) 

TCi  -  The  cost  to  train  one  individual  of  a  specified  MOS  through  a  specified 
sequence  of  courses  (the  cost  per  graduate,  taking  attrition  into 
account)  using  a  specified  training  regimen  and  suite  of  associated 
equipment 


Elements  of  TCi  (the  long  run  average  cost  to  train  an  individual  through  a 

particular  sequence  of  courses)  include  the  following: 

1 )  Student  costs  (time  to  train  multiplied  by  pay  and  allowance  rate) 

2)  Travel,  food,  and  lodging 

3)  Instructor  costs 

4)  Equipment  costs 

5)  Facilities  costs 

6)  Other  related  costs 

7)  New  hardware,  software,  and  courseware,  if  appropriate. 

NOTES: 

1 )  This  is  the  simple  formal  training  cost. 

2)  The  course  sequence  will  address  the  acquisition  of  a  specific  skill  set  (specialty), 
usually  for  a  single  rank  or  a  narrow  band  of  ranks. 

3)  We  may  be  aggregating  across  several  courses  in  a  sequence. 

4)  Course  attrition  must  be  taken  into  account. 

5)  Do  not  consider  previous  R&D  investments  to  develop  the  current  training  regimen  or 
the  new  training  technology  (we  are  considering  only  off-the-shelf  state-of-the-art). 

6)  Assume  no  new  ’costs  of  techrwiogy’  (i.e.,  no  new  R&D;  we  are  applying  the  state- 
of-the  art,  not  advancing  it). 

7)  These  costs  may  be  divided  among  several  course  sequences  in  which  the  same 
new  hardware,  software,  and  courseware  are  put  to  use. 

0)  Nonstandard/specialized  hardware  and/or  software  may  be  necessary  for  a 
particular  course  sequence. 

9)  Time  to  train  and  associated  costs  (e.g.,  number  of  instructors)  may  decrease, 
leading  to  an  overall  cost  decrement 

10)  All  new  hardware,  software,  and  courseware  costs  must  be  suitably  amortized  over 
time. 


I 
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4.  Step  4:  Determine  the  Training  Effectiveness  of  Each  Option 

We  now  face  the  difficult  question  of  training  effectiveness,  that  is,  the  incremental 
gain  in  proficiency  provided  by  each  training  option.  Let  us  refer  to  this  quantity  as  the 
training  effectiveness  for  option  i,  TEi.  Whether  the  training  options  are  teaching  new 
skills  for  the  first  time  or  providing  for  the  augmentation  or  remediation  of  old  skills,  it  is 
important  to  measure  the  trainee's  ability  to  perform  the  requisite  tasks  both  prior  to  and 
subsequent  to  the  training  options  under  analysis.  In  some  cases,  the  a  priori  testing  may 
be  waived  because  the  student  may  simply  have  no  ability  to  perform  the  particular 
specialized  tasks  before  the  subject  training. 

What  we  seek  here  are  performance-oriented  metrics  that  can  be  applied  pre-  and 
post-training  and  that  can  be  related  to  proficiency  in  performing  the  skills  in  question. 
These  measures  must  assure  that  the  data  they  produce  are  at  an  interval-scale  level  of 
measurement,  or  can  be  converted  to  such  a  scale,  so  that  differences  can  be  included  in  a 
cost-effecdveness  ratio.  It  should  be  noted  that  the  practice  of  fixing  either  cost  or 
effectiveness  in  training  cost-effectiveness  analyses  typically  seen  in  the  literature  is  both 
arbitrary  and  unnecessary  if  one  assumes  that  both  cost  and  effectiveness  can  be  measured 
on  at  least  an  interval  scale,  as  we  have  required  in  our  methodology.  In  addition,  the 
measures  must  depend  on  data  that  already  exist  or  can  be  derived  and  must  assure  that  the 
data  be  available  pre-  and  post-training.  In  fact,  the  best  data  we  might  have  are  fitness 
reports  before  and  after  training.  In  the  final  analysis,  however,  we  must  be  capable  of 
measuring  skills  relevant  to  performing  the  tasks  in  question  or  possess  the  ability  to 
measure  performance  on  the  specific  tasks  in  question  for  which  such  skills  will  be  put  to 
use. 

Applying  the  pre-  and  post-training  performance  metrics  in  the  context  of  new 
technology  may  be  difficult  since,  unfortunately,  it  is  not  always  possible  to  give  such 
technology  a  preliminary  trial.  The  cost  of  developing  the  necessary  courseware  and 
applying  it  on  a  limited  basis  can  approach  50  percent  or  more  of  what  it  would  cost  to 
apply  it  to  all  the  students  in  a  training  sequence.  If  pilot  tests  are  not  appropriate,  from 
where  do  we  derive  these  data?  First,  we  turn  to  the  data  on  training  effectiveness  that  we 
can  deduce  from  similar,  previous  applications  of  training  methodologies  and  technologies. 
We  ought  to  find  data  on  proficiency  increments  resulting  from  such  previous  applications. 
If  these  do  not  exist,  we  could  look  at  data  fn-  other  related  technology  enhancements  in  the 
past,  for  example,  the  introduction  of  CBI  in  the  1970s  and  early  1980s. 
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A  good  source  of  the  necessary  data  is  the  research  literature,  but  the  data  will  have 
to  be  interpreted  and/or  extrapolated.  For  example,  there  is  the  possibility  that  a 
qualitatively  different  training  effect  is  inherent  in  the  new  technology  applied  in  one 
training  option  and  not  provided  by  the  older  technology  in  another  option.  Similarly,  the 
existing  literature  suggests  that  it  takes  30  percent  less  time  to  train  to  criterion  using  IVD 
technology  versus  existing  training  methods  (Retcher,  1989, 1990).  How  such  statistics 
translate  into  increased  proficiency  is  not  clear,  but  an  effort  at  such  translation  is  necessary 
for  the  success  of  our  methodology. 

We  must  then  convert  the  estimates  of  the  training  effectiveness  of  each  option  to 
common  interval  scale  measures  of  student  average  proficiency  at  performing  the  requisite 
tasks/duties/skills.  We  will  need  measures  of  student  average  proficiency  gains  at 
performing  the  requisite  tasks  for  each  option,  taking  speed  and  accuracy  into  account. 
Specifically,  we  will  require  data  on  average  training  effect  size  due  to  the  alternative 
training  regimens  (from  the  research  literature  if  necessary)  for  the  appropriate  student 
population  and  performance  area.  The  following  measures  could  all  prove  useful  in 
computing  the  tinal  interval  scaled  quantity  necessary  for  our  analysis: 

1 .  Knowledge  and  performance  traiiting  outcomes  (paper  and  performance  tests) 

2 .  Retention  of  trained  skills  and  learned  knowledge 

3 .  Post-training  fimess  reports  (or  estimates)  for  each  training  option 

4 .  Time  on  task  during  training  (secondary  measure) 

5 .  Time  to  complete  instructicm  (secondary  measure) 

6 .  Time  to  complete  performance/knowledge  tests  (secondary  measure). 

Once  all  of  the  above  data  are  gathered,  we  must  aggregate  them  using  a  function 
we  will  develop  for  converting  these  measures  of  student  proficiency  gains  to  an  interval 
scale.  The  form  of  this  aggregating  function  depends  on  the  relationship  between 
performance  measures  and  Icmg-teim  military  proficiency.  However,  we  may  only  need  to 
weight  appropriately  (in  a  subjective  manner)  the  various  measures  in  order  to  combine 
them.  For  this  reason  the  aggregation  may  be  linear  or  nonlinear  and  may  require  us  to 
view  the  individual  proficiency  measures  as  independent  or  related.  This  could  involve 
some  nonlinear  transformation  of  one  or  more  of  the  scales  so  that  ratios  of  differences  on 
any  one  (and  therefore  a  weighted  average  of  all  measures)  will  p:ove  meaningful.  For 
example,  a  10-point  improvement  on  post-training  paper  or  performance  tests  may  not  be 
worth  half  as  much  as  a  20-point  improvement  on  the  same  tests.  It  may  be  worth,  for 
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example,  two-thirds  as  much  in  terms  of  its  implications  for  long-term  military  proficiency 
or  ability.  In  combining  measures  into  our  final  unidimensional  interval  scale  measure,  it 
will  be  important  to  take  their  nonindependence  into  account. 

A  final  point  to  be  made  with  respect  to  determining  the  proper  measures  to  use  in 
our  analysis  is  that,  ideally,  our  measure  of  the  effectiveness  of  training  would  relate  more 
to  the  military  value  of  training  (for  example,  its  contribution  to  readiness)  than  to  the 
improvement  in  individual  performance.  For  now,  however,  we  will  not  attempt  to 
address  or  even  to  raise  this  difficult  issue.  We  will  simply  comment  that  measurement  of 
contributions  of  training  to  readiness  and,  ultimately,  success  in  battle  should  be  the  goal 
for  which  we  strive. 

It  should  be  noted  that  some  of  the  difficulty  in  measuring  the  differential  training 
effectiveness  of  training  options  might  be  obviated  by  the  emphasis  on  training  to  criterion. 
That  is,  we  may  be  able  to  assume  that,  under  any  option  involving  new  or  enhanced 
training  technology,  no  increase  in  proficiency  will  occur  since  the  same  standard  set  of 
criteria  are  used  to  determine  when  the  degiee  of  training  has  been  sufficient.  This 
assumption  may  greatly  simplify  our  methodological  approach.  Table  in-4  summarizes  the 
key  considerations  for  computing  the  quantity  TEi.  (Sec  Appendices  B  and  C  for  a 
summary  of  the  activities  and  infemnation  items  required  for  this  step.) 

5.  Step  5:  Perform  Cost-Effectiveness  Analysis  and  Draw  Conclusions 

With  the  data  generated  by  implementation  of  the  first  four  steps  of  the  TFDM,  it  is 
possible  to  compute  and  compare  the  ratios  of  training  effectiveness  (the  increment  in 
student  proficiency)  to  the  cost  per  student  for  each  training  option.  If  o:  e  op  on  provides 
a  higher  ratio  of  incremental  productivity  per  dollar,  then  it  is  the  option  of  choice, 
assuming  it  meets  budget  constraints  and  that  there  are  no  significant  nonmonetary 
arguments  to  the  contrary.  We  will  be  interested  in  the  percentage  difference  in  cost- 
effectiveness  between  options. 


Table  111-4.  Determine  the  Training  Effectiveness  of  Each  Option  (TEI) 

Training  Effectiveness  i  =  The  increase  in  a  single  student's  proficiency/ability  at 

performing  the  tasks/duties/skills  for  which  a  particular 
training  option  is  designed  to  provide  training 

Proficiency  =  An  interval  scale  measure  of  a  student's  ability  to  perform 

a  designated  set  of  tasks  which  takes  into  account 
accuracy  and  speed  (i.e.,  an  interval  scale  measure  of 
accomplishment  per  unit  time) 

Sources  of  Data:  1)  Similar  existing  training  applications  for  options  of  the 

new,  modified,  or  expanded  involving  new  training 
methodology/technotogy 

2)  Past  training  applications  of  similar  training  tech¬ 
nologies 

3)  The  research  literature 

4)  Common  sense  and  guesswork. 

NOTES: 

1 )  In  general,  a  single  course  sequence  will  train  one  rank  or  a  narrow  band  of  ranks  to 
perform  the  duties  of  a  single  MOS  or  several  associated  MOSs. 

2)  Ratios  of  differences  in  proficiency  (i.e.,  training  effectivenesses)  are  appropriate 
for  use  in  an  effectiveness/cost  analysis. 

3)  The  value  of  the  qualitatively  different  training  provided  through  any  new  training 
technology  must  be  taken  into  account. 

4)  Increased  speed  in  task  performance  as  a  resuH  of  training  is  a  critical  consideration 
for  training  options  involving  few  technologies. 

5)  Even  if  training  is  continued  only  until  a  pre-defined  set  of  performance  criteria  are 
met,  there  may  wall  be  an  increase  in  productivity  resulting  from  the  new  training 
regimen/procedures. 


To  illustrate  the  above,  let  us  assume  that  we  have  taken  a  multimethod  approach  to 
measuring  proficiency  and  have  developed  a  scale  (some  linear  combination  of  nonlinear 
functions  of  other  measures  taken  on  other  scales)  that  measures  an  individual's  ability  to 
perform  his  necessary  tasks  and  dudes  on  an  absolute  interval  scale  from  0  to  100.  Prior  to 
the  current  training  sequence,  the  individual  may  perform  at  a  proficiency  level  of  50 
(which  we  will  assume  to  be  unacceptable)  on  the  requisite  skills,  tasks,  or  duties.  After 
training,  the  individual's  score  may  be,  for  example,  89,  which  we  consider  quite  good 
relative  to  our  highest  score  of  100.  We  may  also  have  determined  that  the  average  cost  to 
train  an  individual  through  this  segment  of  the  training  pipeline  is  $15,000.  We  therefore 
have  a  ratio  of  39  proficiency  units  to  $15,000,  or  2.6  units  of  individual  proficiency  per 
$1,000. 

Given  the  cost  and  proficiency  measures  that  have  been  described  above,  we  can 
calculate  the  necessary  ratios  of  training  effectiveness  divided  by  cost  and  perform 
comparisons  of  effectiveness/cost  ratios  and  other  key  factors.  We  will  be  interested  in  the 
percentage  difference  in  cost-effectiveness  between  options.  We  will  also  be  interested  in 
comparing  the  number  of  trainees  through  the  pipeline  per  unit  time  and  forcewide 
increases  in  proficiency.  Finally,  we  will  want  to  compare  the  total  costs  of  training  for  all 
trainees  who  can  be  trained  in  a  fixed  period  of  time. 

♦ 

6 .  Summary 

We  have  provided  a  general  overview  of  our  five-step  TFDM  procedure.  Our  goal 
is  that  it  serve  as  a  general  guide  for  the  case  study  we  will  now  develop.  The  case  study 
involves  the  application  of  CBI/IVD  training  techniques  and  equipment  to  electronics 
maintenance  training.  This  application  should  assist  us  by  providing  the  additional  insights 
necessary  to  "add  meat  to  the  bones"  of  the  general  TFDM  outlined  above.  It  is  also 
possible  that  the  results  of  this  "pilot  test"  of  the  TFDM  will  prove  valuable  in  their  own 
right  in  the  context  of  today's  resource  allocation  decisions. 
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IV.  APPLYING  THE  TRAINING  FUNDS  DECISION 
METHODOLOGY  TO  A  CBI/IVD  RESOURCE  ALLOCATION 
DECISION:  A  SAMPLE  CASE  STUDY 


This  section  presents  an  illustrative  case  study  demonstrating  the  application  of  the 
Training  Funds  Decision  Methodology  (TFDM).  In  it  we  address  the  following  question; 
"Should  support  for  computer-based  instruction/interactive  videodisc  technology 
(CBI/rVD)  for  electronics  maintenance  training  be  increased  by  a  speciHed  amount?"  Such 
an  inquiry  assumes  that  the  general,  possible,  and  realistic  order  of  magnitude  of  such  an 
increase  is  known  in  advance.  This  increase  will  be  some  quantity  large  enough  to  have  a 
foreseen  positive  impact  on  the  effectiveness  of  electronics  maintenance  traimng,  but  small 
enough  to  comply  with  realistic  budget  constraints. 

A  signiHcant  amount  of  up-front  analysis  would  have  to  be  performed  to  determine 
the  proper  magnitude  of  this  increase  and  to  assure  that  it  represents  an  increase  large 
enough  to  truly  have  a  positive  impact  on  electronics  maintenance  training.  This  does  not 
imply  the  need  for  a  complete  analysis  of  hardware,  software,  and  courseware  that  could  be 
bought  with  the  specified  increase,  only  that  the  increase  be  seen  as  large  enough  to  merit 
the  effort  of  a  more  thorough  analysis.  Indeed,  an  informal  cost-benefit  analysis  should 
ideally  be  undertaken  a  priori  to  identify  the  best  candidate  technologies  for  consideration  to 
receive  funding  increases  and  to  detemune  the  appropriate  orders  of  magnitude  for  such 
contemplated  increases. 

We  will  now  proceed  with  two  versions  of  a  CBI/IVD  case  study  for  electronics 
maintenance  training.  The  fundamental  difference  between  the  two  is  that,  in  the  first,  both 
acquired  proficiency  and  cost  of  training  change;  in  the  second,  training  time  alone 
changes.  The  same  proficiency  level  is  attained  in  both  the  original  (i.e.,  current)  and  the 
modiHed  (i.e.,  to  include  more  CBI/IVD  technology)  training  regimens.  As  previously 
noted,  however,  there  are  likely  to  be  certain  difficulties  in  measuring  changes  in  the 
proficiency  with  which  a  soldier  will  perform  his  electronics  maintenance  specialty.  This  is 
due  especially  to  the  fact  that  increases  in  training  funds  generally  tend  to  go  toward 
systems  that  provide  more  realistic  environments  in  which  to  train.  Such  is  the  case  with 


CBI/rVD,  which  leads  to  greater  performance  proficiency  in  the  anticipated  actual  wartime 
setting. 

A.  VERSION  1  OF  THE  CBI/IVD  ELECTRONICS  MAINTENANCE 
TRAINING  CASE  STUDY 

Let  us  assume  that  a  $125,000  training  augmentation  is  being  considered  which  will 
provide  for  the  purchase  of  new  CBI/IVD  hardware,  software,  and  courseware  to  train  a 
specific  MOS  involving  the  repair  and  maintenance  of  electronics  equipment  The  question 
to  be  addressed  is,  "How  cost-effective  would  such  an  investment  in  new/enhanced 
training  technology  be,  compared  with  the  alternative  existing  training  regimen?"  To 
determine  whether  this  is  a  wise  investment  we  will  proceed  through  the  five  steps  of  the 
TFDM  as  described  in  the  previous  chapter. 

1 .  Step  1:  Frame  the  Question 

Here  we  match  the  CBI/TVD  analysis  situation  with  a  similar  analysis  template 
chosen  from  a  predefined  taxonomy  of  possible  choices.  In  terms  of  the  seven  components 
in  the  Gibson  et  al.  (1988)  taxonomy  of  training  policy  issues,  our  question  can  be 
analyzed  as  follows: 

1 .  The  question  addresses  individual  training. 

2 .  Performance  will  be  allowed  to  vary  (in  the  first  version  of  the  analysis). 

3 .  Training  time  may  vary. 

4 .  Training  volume  may  vary. 

5 .  An  innovative  training  technology  is  concerned. 

6.  Course  design  may  change. 

7 .  Location  is  not  likely  to  change  and  new  facilities  are  unlikely  to  be  necessary. 

In  terms  of  the  components  of  the  Canadian  (1987)  taxonomy,  we  observe  the 
following  characteristics  represented  in  our  question: 

1 .  This  is  a  relatively  new  method  of  instruction. 

2.  We  are  addressing  one  new  training  resource,  not  a  mix. 

3 .  We  will  lo(^  at  individual  peifoimance. 

4.  Our  focus  will  be  on  technology  but  involve  individual  courses  and  a  particular 
specialty  area. 


5 .  We  are  not  trying  to  optimize  a  mix  of  training  resources. 

6.  We  are  not  looking  at  ease  of  training  for  a  new  system. 

7 .  We  will  be  implementing  and  evaluating  (to  make  our  choice)  a  new  method, 
but  not  developing  it. 

8.  Ideally,  we  will  be  focusing  on  the  tasks  to  be  trained  and  associated 
performance  objectives. 

9.  We  are  not  evaluating  a  training  organization  or  infrastructure;  however, 
CBI/IVD  may  require  some  change  in  these. 

10.  We  will  have  to  analyze  training  research  data  because  of  a  lack  of  applications 
data  on  CBI/IVD. 

1 1 .  We  will  not  be  making  training-related  system  design  decisions. 

In  terms  of  Table  II-2  relating  decision  contexts  to  cost  elements  and  effectiveness 
levels,  this  question  for  analysis  best  matches  Column  1,  "Evaluation  of  new  method  of 
instruction  for  a  course."  Therefore,  initial  investment  and  direct  operating  costs  will  be 
more  important  than  R&D,  Support,  Training  Infrastructure,  and  Training  Resource  costs. 
Attainment  of  course  performance  objectives  will  be  the  best  measure  of  training 
effectiveness  levels  available  to  us. 

2 .  Step  2:  Make  the  Options  Explicit 

We  have  already  established  that  an  investment  of  $125,000  is  being  contemplated 
for  the  purchase  or  development  of  new  hardware,  software,  and  courseware  to  be  used  in 
a  typical  four-course  sequence  to  train  a  specific  MOS  involving  the  repair  and  maintenance 
of  electronics  equipment  We  estimate  that  this  will  pay  for  10  video  disc  systems  costing 
$10,000  each,  as  well  as  the  courseware  and  software  development  costs  of  $25,000.  Let 
us  assume  that  there  is  already  some  modest  CBL  as  well  as  actual  electronics  equipment 
workstations,  involved  in  the  course  sequence  as  currently  taught  The  new  CBI/IVD- 
based  training  would  replace  the  existing  CBI-based  instruction,  using  a  small  portion  of 
the  existing  computer  hardware  and  some  of  the  software.  The  old  courseware  would 
become  obsolete.  Let  us  also  assume  that  we  are  looking  at  the  next  5  years  as  the  useful 
lifetime  of  the  new  training  technology.  Finally,  let  us  assume  that  in  any  one  of  the 
5  years,  an  average  of  120  trainees  will  progress  through  the  sequence  with  the  new 
equipment,  as  we  discussed  above,  starting  with  100  in  Year  1  and  linearly  increasing  to 
140  in  the  fifth  and  final  year. 


3.  Step  3:  Determine  the  Costs  of  Options 

Remember  that  we  are  seeking  to  determine  the  cost  to  train  one  individual  using  the 
four-course  sequence  of  electronics  maintenance  courses  for  this  MOS,  flrst  making  no 
changes  and  next  introducing  the  new  technology.  The  courses  will  be  substantially 
modified  as  a  result  of  the  subject  CBI/IVD  technology  insertion.  It  should  be  noted  that 
the  contemplated  investment  could  actually  affect  more  than  one  MOS  and  several 
sequences  of  courses.  If  so,  we  would  need  to  perform  this  analysis  for  each  and 
aggregate  all  costs  and  proficiency  gains  appropriately.  However,  for  the  purpose  of  this 
case  study,  we  will  stay  within  the  narrower  context 

Once  we  have  determined  the  course(s)  or  course  sequence(s)  affected,  we  must 
determine  the  average  long  run  cost  to  train  an  individual  using  them.  Since  there  are  four 
such  courses,  we  will  be  aggregating  student  costs,  travel,  food  and  lodging  costs, 
instructor  costs,  equipment  and  facilities  costs,  and  other  related  costs  as  identified  by 
Orlansky  and  String  (1979).  Let  us  assume  that  the  particular  four-course  sequence 
affected  involves  courses  taught  in  two  separate  1-  month  visits  to  a  training  school  with 
each  visit  separated  by  6  months  and  that  the  average  cost  for  this  training  is  $25,000  per 
student,  taking  all  costs  into  account.  The  old  equipment  has  allowed  us  to  train  100 
students  per  year,  but  we  expect  to  increase  the  number  of  trainees  to  110  in  the  second 
year  of  the  new  equipment,  120  in  the  third,  etc.  For  our  example,  assume  that  the  current 
training  regimen  costs  break  out  as  follows:  $5,000  in  student  costs;  $5,000  in  trav^foot^ 
and  lodging  costs;  $5,000  in  instructor  costs;  $5,000  in  equipment  and  facilities  costs;  and 
$5,000  in  other  related  costs.  A  data  base  designed  to  handle  data  such  as  these  is  under 
development  at  TPDC.  It  should  be  noted  that  the  assumed  costs  above  are  somewhat 
arbitrary  and  were  chosen  to  make  the  case  study  easy  to  understand.  They  are 
representative  of  historical  case  data  surveyed  by  the  authors,  but  not  meant  to  replicate  any 
one  particular  case  study.  Our  primary  purpose  in  presenting  these  numbers  is  to 
demonstrate  the  operations  in  the  TFDM. 

We  must  now  determine  the  sum  of  the  costs  per  student  for  new  hardware, 
software,  and  courseware,  as  well  as  all  related  costs  as  above  identified.  It  should  be 
noted  that  the  related  costs  are  likely  to  decrease  as  a  result  of  the  technology  investment. 
For  example,  student  time  and  instructor  time  are  likely  to  decrease,  the  former  by 
approximately  30  percent  and  the  latter  by  40  percent  (Fletcher,  1990).  The  first  critical  step 
is  to  determine  exactly  what  CBI/IVD  hardware,  software,  and  courseware,  together  with 
any  necessary  nonstandaid/specialized  hardware  and/or  software  specifically  for  electronics 


maintenance  training  will  be  purchased  for  our  $125,000  investment  and  how  many 
individuals  this  will  be  used  to  train  over  what  period  of  time.  We  have  already  determined 
most  of  the  required  data  from  our  front-end  analysis  in  Step  2,  above.  Second,  we  will 
need  to  project  exactly  how  the  existing  training  regimen  will  change  as  a  result  of  the  new 
equipment,  particularly  with  respect  to  duration  and  instructor  needs,  as  these  changes  will 
have  a  major  effect  on  any  savings  we  might  anticipate  as  a  result  of  the  up-front 
investment.  Let  us  first  assume  that  $12,500,000  would  typically  be  used  to  train  500 
individuals  over  the  next  5  years,  at  which  point  the  next  generation  of  electronics 
maintenance  training  equipment  will  come  on  line.  The  new  equipment  and  software  alone, 
then,  will  cost  approximately  $125,000/600  =  $208.33  per  individual  trained.  If  there 
were  no  savings  anticipated  as  a  result  of  this  new  investment,  the  average  cost  to  train  an 
individual  would  rise  to  $25,000  +  $208.33  =  $25,208.33.  The  total  cost  to  train  the  now 
600  students  would  be  $15,124,998. 

Let  us  assume  a  savings  in  training  time  of  $1,500  (30  percent  x  $5,(XX)),  a  savings 
in  instructor  costs  of  $2,(X)0  (40  percent  x  $5,(XX)),  and  a  savings  in  travel  and  lodging 
costs  of  $500.  Therefore,  costs  to  train  will  decrease  from  the  current  $25,0(X)  per 
individual  to  $21,0(X)  per  individual,  averaged  over  the  next  5  years.  For  each  year  taken 
individually,  the  costs  are  anticipated  to  be  $23,0(X),  $22,0(X),  $21,0(X),  $20,(M)0,  and 
$19,000,  decreasing  with  time,  as  experience  with  the  new  equipment  increases.  As 
implied  above,  this  decrease  is  due  roughly  to  anticipated  new  costs  (averaged  over  the  next 
5  years)  as  follows:  $3,500  in  student  costs;  $4,500  in  travel,  food,  and  lodging  costs; 
$3,000  in  instructor  costs;  $5,000  in  other  equipment  and  facilities  costs;  and  $5,000  in 
other  related  costs.  The  cost  for  the  modified  training  regimen  is  therefore  $208.33  (new 
equipment  investment)  +  $21,000  =  $21,208.33,  a  decrease  of  15.2  percent  in  the  cost  to 
train  an  individual.  Let  us  not  forget  that  we  are  projecting  that  we  will  train  120  students 
each  year  on  the  average  versus  the  cturent  100,  an  increase  of  20  percent  in  the  number  of 
graduates.  Over  the  5  years  we  will  therefore  spend  600  x  $21,208.33  =  $12,724,998,  an 
increase  of  only  $224,998  (1.8  percent)  to  train  100  more  soldiers.  We  now  move  on  to 
the  most  difficult  and  critic? ;  jtep  in  our  analysis,  a  determination  of  the  training 
effectiveness  of  each  option.  See  Table  IV- 1  for  a  summary  of  the  key  elements  of  this 
step. 


Table  IV*1.  Summary  of  Version  1  of  the  CBI/IVD  Case  Study 


Step  3.  Determine  the  Training  Cost  of  Each  Option  (TCI) 

Assumptions;  current  training  cost  as  shown. 

TC1  (student  costs)  +  (travel,  food,  and  lodging  costs)  +  (instmctor  costs) 

+  (equipment  and  facilities  costs)  +  (other  related  costs) 

=  $5,000  +  $5,000  +  $5,000  +  $5,000  +  $5,000 
»  $25,000 

TC2  =  average  cost  per  student  for  the  new  equipment  +  average  of  all  other 
costs  per  student 

s  total  cost  for  the  new  equipment/total  number  of  students  to  be  trained 
over  the  lifetime  of  the  new  equipment  +  adjusted  costs  from  above 
»  $125,000/600  +  $3,500  +  $4,500  +  $3,000  +  5,000  -i-  $5,000 
«  $21 ,208.33  (a  decrease  of  15.16%) 


Step  4.  Determine  the  Training  Effectiveness  of  Each  Option  (TEI) 

TE1  -  (Average  Proficiency  after  the  Current  Training  Regimen)-(Average 
Proficiency  before  the  Current  Training  Regimen)  -  85  -  25  «  60 

TE2  -  89  -  25  -  64  (an  increase  of  6.66%) 

Note  that  these  numbers  are  on  a  zero  (0)  to  100  proficiency/ability  scale  and  may 
be  the  result  of  our  best  possible  estimates  and  inferences.  They  take  speed 
and  accuracy  into  account.  They  are  typicaRy  not  directly  measured  at  this  time. 

Step  5.  Perform  a  Cost-Effectiveness  Analysis 

Calculate  and  compare  the  two  ratios,  TEI/TCI  and  TE2/TC2. 

TE1/TC1  -  60/$25,000  -  2.4/$1 ,000  -  0.0024/$1  (units  of  proficiency  gained 
per  individual  per  $1  spent) 

TE2/TC2  -  64/$21 ,208  -  3.01 8/$1. 000  -  0.00301 8/$1 


4 .  Step  4;  Determine  the  Training  Effectiveness  of  Each  Option 

The  best  measures  we  currently  have  available  from  which  to  estimate  the  gains  in 
job  performance  productivity/proficiency  are  tests  given  during  and  at  the  end  of  the 
training  sequence  and  post-training  fimess  reports  that  focus  on  the  specific  skills  for  which 
the  individual  has  been  trained.  We  typically  do  not  have  a  baseline  (pre-training)  measure 
of  proficiency  since,  prior  to  receiving  training  on  a  skill  set,  individuals  are  rarely  tested, 
evaluated,  or  rated  on  their  abilities  to  perform  that  skill  set  We  may  find  ourselves  having 
to  make  an  assumption  about  the  baseline  proficiency  for  the  particular  population  of 
individuals  trained  for  the  specified  MOS  through  the  subject  course  sequence.  Testing  a 
small  sample  of  the  subject  population  for  their  baseline  proficiency  in  the  required  set  of 
skills  prior  to  receiving  the  subject  course  sequence  is  a  possible  solution  to  this  dilemma, 
but  not  one  that  we  will  propose  for  this  case  study. 

Let  us  assume  that  we  have  constructed  an  interval  scale  that  measures  proficiency 
in  performing  the  required  electronics  maintenance  skills,  taking  both  accuracy  and  speed 
of  performance  into  account.  Our  best  estimate  is  that  prior  to  the  relevant  training 
sequence  an  individual  would  score  approximately  25  on  this  scale,  i.e.. 

Prior  Proficiency  =  f  (Aptitude  Test  Scores,  Fimess  Reports)  =  25. 

This  estimate  is  based  on  pre-training  aptitude  testing  and  fimess  reports  which 
were  used  to  select  the  individuals  for  training.  Measures  taken  from  (1)  tests  administered 
during  and  after  training,  (2)  follow-up  testing,  and  (3)  fimess  reports,  when  combined 
functionally  after  training  under  the  current  regimen,  result  in  a  score  of  85  on  this  scale, 
i.e.. 

Posterior  Profiency  =  f  (Scores  During  Training)  (Scores  After  Training)  =  85. 

Hence,  the  current  training  regimen  provides  on  the  average  a  gain  of  approximately 
60  proficiency  points,  at  an  expense  of  $25,000  per  individual.  That  is. 

Proficiency  Gain  =  Posterior  Proficiency  -  Prior  Proficiency  =  85  -  25  =  60. 

We  must  now  determine  the  likely  increase,  as  a  result  of  the  modified  training 
regimen,  in  the  average  smdent's  proficiency  at  performing  the  electronics  maintenance 
tasks  that  the  specified  four-course  sequence  is  designed  to  train.  Unfortunately,  as 
discussed  above,  this  increase  may  only  be  based  on  an  intelligent  guess.  But  it  can  be  a 
highly  informed  guess  supported  by  data  from  similar  existing  training  applications  of 


CBVIVD  technology.  We  can  also  use  past  training  applications  of  similar  training 
technologies  and  the  research  litCTature  to  assist  us. 

Let  us  assume  that  avenues  of  analysis  have  led  us  to  certain  estimates  of  the 
outcomes  of  the  same  measures  listed  above  (i.e.,  test  scores  during  training,  test  scores 
post-training,  follow-up  test  scores,  fimess  reports)  and  that  when  functionally  combined, 
we  find  that  the  modified  training  regimen  would  produce  students  who  scored  89  on  our 
proficiency  scale.  This  is  an  increase  of  4  points  over  the  score  of  85  for  students  receiving 
the  current  electronics  maintenance  training  regimen.  That  is. 

Posterior  Proficiency  =  f  (same  4  measures)  =  89. 

Under  the  new  training  regimen  there  will,  therefore,  be  a  proficiency  gain  of  64 
points  versus  the  current  gain  of  60,  an  increase  of  slightly  more  than  6.66  percent  in 
proficiency.  That  is. 

Proficiency  Gain  =  Posterior  Proficiency  -  Prior  Proficiency  =  89  -  25  =  64. 

In  support  of  this  hypothetical  statistic,  we  note  that  Fletcher  (1990)  has 
documented  an  average  efiiect  size  of  approximately  .40  standard  deviations  across  24 
studies  of  IVD  applications  to  military  training.  This  irr^lies  an  increase  for  50th  percentile 
students  to  about  the  65th  percentile.  If  we  assume  a  somewhat  arbitrary  standard 
deviation  of  10  for  the  distribution  of  proficiency  gains  (not  unlike  what  Fletcher  found  in 
his  review  of  the  literature),  this  implies  an  effect  size  of  0.4  x  10  =  4  and  60  +  4  =  64.  It 
should  be  noted  that  Fletcher's  effect  size  was  based  on  a  variety  of  measures  including  test 
scores  and  performance  ratings.  We  have  generalized  this  increase  in  effect  size  to 
proficiency.  See  Table  IV- 1  for  a  summary  of  the  key  elements  of  this  step. 

5 .  Step  5:  Perform  a  Cost>EfTectiveness  Analysis  and  Draw  Conclusions 

We  have  now  reached  the  final  step  in  our  TFDM  where  we  must  take  the  results  of 
the  two  previous  steps  and  combine  them  in  a  manner  that  will  allow  us  to  compare  the  two 
resulting  proficiency/cost  ratios.  We  accomplish  this  in  three  steps. 

•  The  ratio  for  the  current  training  regimen  is  60  proficiency  points  divided  by 
$25,000  =  2.4/$ 1,000  =  0.0024/$!  proficiency  gain  per  dollar. 

•  The  ratio  for  the  new  training  regimen  is  64  divided  by  $21,208  = 
3.018/$1,000  =  0.003018/$!  proficiency  gain  per  dollar. 


•  Therefore,  according  to  this  analysis,  it  is  25.75  percent  more  cost  effective  to 
make  the  investment  in  new  technology  over  the  subject  5-year  period, 
assuming  no  new  costs  would  be  incurred  to  stay  with  the  cunent  regimen. 

If  the  current  actual  equipment  trainers  were  in  need  of  replacement  or  even 
upgrade,  the  cost-effectiveness  would  be  even  greater.  There  are  several  additional 
considerations  to  be  factored  into  our  final  decision. 

Under  the  new  CBI/IVD  training  regimen,  the  Army  will  be  able  to  train  a  projected 
600  men  at  a  cost  of  $12,724,800  for  a  gain  of  38,400  proficiency  units  over  the  next  5 
years.  To  leave  training  unchanged  would  cost  $12,500,000  to  train  only  500  men  and 
gain  30,000  proficiency  units.  Hence,  for  1.8  percent  more  dollars,  the  Army  would  get 
an  overall  increment  of  28  percent  in  forcewide  units  of  proficiency. 

What  the  Army  learns  through  this  early  application  of  a  new  technology  would 
clear  the  way  in  terms  of  new  developments  and  lessons  learned  for  future  similar 
applications.  There  is  simply  some  up-front  investment  to  be  made  in  order  to  derive  such 
benefits.  (See  Table  IV- 1  for  a  summary  of  the  key  elements  of  this  final  step.)  Therefore, 
from  a  cost-effectiveness  perspective,  the  investment  in  the  new  CBUTVD  equipment  and 
the  modification  of  the  current  training  regimen  to  suit  it  is  a  sound  investment  In  addition 
to  the  increased  proficiency,  there  are  other  intangible  benefits  that  are  sure  to  accrue. 

B .  VERSION  2  OF  THE  CBI/TVD  ELECTRONICS  MAINTENANCE 
TRAINING  CASE  STUDY:  PROFICIENCY  CONSTANT 

This  second  version  of  the  case  study  in  similar  to  the  first  except  that  we  assume 
that  the  proficiency  gain  under  the  new  CBI/TVD  training  regimen  is  held  constant 
compared  to  the  old  since  we  are  training  to  a  performance  level  where  the  same  specific  set 
of  performance  criteria  will  be  met.  That  is,  there  will  be  the  same  60-point  gain  in 
proficiency.  For  this  to  occur,  time  to  train  must  decrease  as  will  all  associated  expenses. 
Assuming  that  proficiency  gain  is  a  negatively  accelerating  function  of  time  to  train,  in 
order  to  gain  only  60  versus  64  proficiency  points  (6.25  percent  less)  under  the  CBI/TVD 
regimen,  we  estimate  that  we  must  decrease  the  time  to  train  by  an  additional  multiple  of 
approximately  10  percent  on  top  of  the  initial  reduction  of  30  percent  We  must  therefore 
reduce  anticipated  student,  instructor,  food  and  lodging,  equipment  and  facilities,  and  other 
related  costs  by  10  percent  The  only  costs  not  affected  would  be  travel  costs  and  CBI/IVD 
equipment  costs,  which  we  will  assume  are  $1,708.33  of  the  estimated  $21,208.33. 


CBI/rVD  costs  would  be  likely  to  decrease  on  a  per  student  basis  since  more 
students  should  move  through  the  training  pipeline  during  the  lifetime  of  the  equipment 
under  a  "training  to  criterion"  regimen  and  each  student  would  take  less  time  to  train.  The 
new  cost  would  therefore  be  $1,708.33  +  ($19,500  x  0.9)  =  $19,258.33.  The  new 
proficiency  gain/cost  ratio  is  60/$19,258.33  =  3.1 16/$1,0(X)  =  0.0031 16/$1.  This  ratio  is 
29.81  percent  greater  than  that  for  the  old  training  regimen  versus  25.75  percent  greater  in 
Version  1.  Hence,  holding  proficiency  gain  constant  is  even  more  cost-effective  than 
allowing  it  to  grow  in  an  unbounded  manner. 

C .  SUMMARY  OF  THE  CASE  STUDY 

We  have  seen  from  the  two  versions  of  the  case  study  that  it  should  be  possible  to 
perform  explicit  training  cost-effectiveness  analyses  for  training  options  which  take  into 
account  appropriate  measures  and/or  estimates  of  the  costs  and  benefits  of  training  under 
various  regimens.  The  gathering  and/or  estimating  of  cost  data  is  an  important  and  difficult 
problem,  but  not  nearly  as  critical  a  problem  as  that  of  estimating  the  effectiveness,  and 
hence  the  value,  of  military  training.  As  a  result  of  this  study  we  have  indicated  the  types 
of  data  that  are  necessary  and  suggested  the  requirements  for  properly  combining  them. 


V.  SUMMARY,  CONCLUSIONS,  AND 
RECOMMENDATIONS 


The  purpose  of  this  paper  has  been  to  outline  a  process  for  utilizing  existing  data 
bases  and,  if  necessary,  developing  new  ones  to  answer  key  policy  questions  regarding 
DoD  expenditures  of  funds  for  various  forms  of  training.  This  project  has  developed  a 
general  cost-benefit  model  called  the  Training  Funds  Decision  Methodology  (TFDM),  for 
conducting  analyses  in  support  of  questions  focused  on  the  desirability  of  iiicreasing, 
decreasing,  or  holding  constant  various  elements  of  DoD  funding  for  training.  Although 
our  methodology  is  illustrated  with  a  specific  case  study,  it  is  designed  to  have  general  and 
wide-ranging  applicability. 

Two  versions  of  the  CBI/TVD  technology  case  study  are  presented,  one  allowing 
effectiveness  of  the  resulting  modified  training  regimen  to  increase  beyond  its  current  level 
and  the  other  holding  the  effectiveness  of  the  modified  training  regimen  constant  The  latter 
is  the  one  most  often  encountered  in  the  real  world  under  the  concept  of  training  to 
criterion.  In  the  first  case,  the  emphasis  was  on  unconstrained  increased  proficiency  in 
performing  the  tasks  for  the  specified  electronics  maintenance  MOS.  In  the  second  case, 
the  emphasis  was  more  on  the  monetary  savings  that  would  result  fix)m  the  modified 
training  regimen  for  electronics  maintenance  repair,  holding  proficiency  constant,  or  as 
nearly  constant,  as  possible  (i.e.,  traiiung  to  the  same  set  of  performance  criteria  under  the 
nxxlified  training  regimen  as  under  the  current  training  regimen). 

The  TFDM  draws  on  general  principles  of  trainiiig  cost  analysis  put  forth  in  detail 
by  previous  authors,  as  well  as  generalized  DoD  cost-effectiveness  analysis  methodologies 
as  developed  and  applied  by  other  practitioners.  The  TFDM  was  designed  to  make  optimal 
use  of  existing  training  cost  (as  well  as  effectiveness)  data  bases.  The  methodology  has 
emphasized  macro-level  simplicity  and  robustness  over  micro-level  detail  and  complexity. 
With  our  methodology  as  a  starting  point,  it  should  now  be  possible  to  add  refinements  and 
augmentations  and  to  address  the  entire  spectrum  of  potential  training  cost-effcctiveness 
questions. 


A.  SUMMARIES  OF  SPECIFIC  CHAPTERS 


1 .  Chapter  II,  Requirements  Analysis 

A  number  of  authors  have  attempted  to  capture  the  essential  elements  of  the  military 
training  policy-making  function  as  performed  by  the  macro-level  training  manager.  In 
Chapter  II,  after  providing  a  succinct  statement  of  the  mission  and  functional 
responsibilities  of  the  DTP  of  the  OASD  for  FM&P,  we  defined  the  role  of  the  macro-level 
training  manager  and  addressed  the  need  for  macro-level  training  data  bases  and  tools  for 
their  use  by  the  macro-level  training  manager. 

Based  on  the  firmly  identified  need  for  macro-level  training  data  bases  and 
techniques  and  tools  for  their  use,  we  put  forth  the  objectives  of  this  particular  paper,  as 
well  as  the  overall  effort  for  which  it  provides  the  final  element  We  proceeded  then  to  lay 
the  foundation  for  this  effort  by  clarifying  the  objectives  and  products  of  two  previous 
efforts  in  this  series  (Gibson  and  Orlansky,  1988,  and  Gibson  et  al.,  1989).  First,  we 
presented  a  model  for  the  conversion  of  resources  to  training  products  which  assumes  that 
the  training  process,  as  guided  by  the  macro-level  training  manager,  converts  money  into 
combat  readiness.  We  went  on  to  present  the  TMM  through  which  the  macro-level  training 
manager  perfonns  his  management  of  the  conversion  process.  The  essential  components 
of  this  model  were  identified  as  requirements,  rerources,  training  activities,  performance 
evaluation  quality  control,  and  readiness  status.  The  structural  components  were  the  cost- 
production  function,  the  performance  effectiveness  function,  and  the  readiness  evaluation 
function. 

In  Chapter  II  we  also  drew  upon  the  work  of  the  Canadian  Bureau  of  Management 
Consulting  (1987)  to  augment  the  woik  of  Gibson  and  Orlansky  (1988)  and  Gibson  et  al. 
(1989).  Specifically,  we  looked  at  the  information  linkages  in  cost-effectiveness  analysis 
of  training,  which  essentially  provided  us  with  a  Canadian  version  of  our  TMM  with  the 
role  of  the  decision-maker  in  evaluating  altemati  ;s  made  more  explicit  and  the  concept  of 
readiness  deleted  from  the  model.  We  went  on  to  focus  on  the  varying  nature  of  the  macro¬ 
level  training  decision  contexts  and  how  this  affects  the  level  of  analyses  of  both  cost  and 
effectiveness  elements.  Finally,  we  examined  the  Canadians'  views  on  providing  for 
efficiency  in  cost-effectiveness  analysis.  They  take  what  we  in  the  United  States  have 
attempted  at  TPDC  a  step  further,  proposing  data  bases  for  cost-effectiveness  analysi 
results  themselves  (not  simply  cost  and  effectiveness  independently)  and  an  independent 
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cost-effectiveness  evaluation  entity  to  perform  all  required  military  cost-effectiveness 
analyses  as  an  impartial  body. 

Finally,  in  Chapter  II,  we  returned  to  a  consideration  of  the  cost  data  elements  we 
must  draw  upon  and  how  they  are  conditioned,  as  the  Canadians  have  noted,  on  the 
particulars  of  the  requirements  and  resulting  type  of  training  under  consideration.  A  careful 
consideration  of  the  appropriate  cost  categories  and  elements  was  seen  to  be  critical  for  the 
success  of  the  training  funds  decision  process  and  the  supporting  training  funds  decision 
methodology. 

2 .  Chapter  III,  The  Training  Funds  Decision  Methodology 

In  Chapter  HI,  we  presented  our  methodology  for  cost-effectiveness  analysis  in 
support  of  the  training  policy-making  function.  First,  we  reviewed  a  number  of  studies  of 
cost-effectiveness  analysis  of  military  training  options.  Next,  we  identified  in  some  detail 
the  training  cost  and  effectiveness  data  requirements  and  data  sources  for  cost-effectiveness 
analysis  of  military  training  options.  Finally,  drawing  upon  previous  material,  we 
presented  our  own  training  funds  decision  methodology  in  support  of  the  macro-level 
decision-maker  for  training  policy. 

In  Chapter  III  we  reviewed  seven  previously  published  studies,  all  of  which 
focused  on  the  use  of  methodologies  for  cost-effectiveness  analysis  of  military  training. 
Several  of  these  studies  were  themselves  lengthy  reviews  of  other  published  studies  of 
cost-effectiveness  analysis.  The  purpose  here  was  to  lay  both  a  theoretical  and  pragmatic 
foundation  for  what  was  to  follow,  based  upon  a  record  of  such  analyses.  The  seven 
studies  reviewed  were: 

•  Cost  and  Effectiveness  of  Computer-based  Instruction  in  Military  Training 
(Orlansky  and  String,  1979) 

•  Cost-Effectiveness  of  Maintenance  Simulators  for  Military  Training  (Orlansky 
and  String,  1981) 

•  Framework  for  Evaluating  the  Cost-Effectiveness  of  Military  Equipment  an  i 
Procedures  (Chatelier  et  al.,  1985) 

•  "Cost-Effectiveness  of  Military  Trairung"  (Orlansky,  1985) 

•  "A  Model  of  Cost/Effectiveness  Analyses  of  Trairung  Devices"  (von  Baeyer, 
1985) 

•  Economic  Issues  in  Cost-Effectiveness  Analyses  of  Military  Skill  Training 
(Solomon,  1986) 


•  Recent  Studies  on  the  Cost-Effectiveness  of  Military  Training  in  TTCP 
Countries  (Fletcher  and  Orlansky,  1989) 

It  was  found  that  the  required  categories  of  cost  data  were  well  understood  and  that 
the  speciHc  requirements  for  the  necessary  training  cost  data  bases  to  make  these  data 
accessible  were  rapidly  taking  shape.  Unfortunately,  only  limited  portions  of  the  required 
data  bases  have  been  implemented  and  plans  for  moving  forward  with  this  critical  need 
were  found  to  be  relatively  sparse. 

Little  has  been  done  during  the  past  decade  to  create  the  necessary  on-line  data 
bases  to  support  cost-effectiveness  analyses  of  training.  With  respect  to  the  modified 
training  cost  data  requirements  and  sources,  we  found  that  we  faced  an  even  more  difficult 
situation  than  we  did  for  existing  training  regimens.  The  categories  of  cost  data  required 
remained  basically  unchanged,  except  for  a  greater  emphasis  on  up-front  research, 
development,  and  acquisition  costs.  Finally,  it  was  found  that  the  data  necessary  for 
comparing  the  effectiveness  of  different  training  regimens  have  been  identified  by  a  number 
of  authors.  However,  few  of  these  data  were  being  systematically  maintained  in  data  bases 
or  even  appeared  to  be  used  systematically  in  published  research  studies. 

Finally,  we  provided  a  general  overview  of  our  five-step  TFDM  procedure.  Our 
goal  was  that  it  serve  as  a  general  guide  for  the  case  study,  with  which  we  then  proceeded. 
The  case  study  involved  the  application  of  dBl/IVD  training  techniques  and  equipment  to 
electronics  maintenance  training.  The  purpose  of  this  application  is  to  provide  the 
additional  insight  needed  to  augment  and  refine  the  general  TFDM  we  had  outlined. 

The  five  steps  in  the  general  methodology  were  the  following:  (1)  Frame  the 
question;  (2)  Make  the  options  explicit;  (3)  Determine  the  costs  of  the  options; 
(4)  Determine  the  training  effectiveness  of  the  options;  (5)  Perform  cost-effectiveness 
calculations  and  draw  conclusions. 

3.  Chapter  IV,  Case  Study 

In  Chapter  IV  we  saw  from  the  two  versions  of  the  case  study  that  it  should  be 
possible  to  perform  explicit  training  cost-effectiveness  analyses  for  training  options  which 
take  into  account  appropriate  measures  and/or  estimates  of  the  costs  and  benefits  of  training 
under  various  regimens.  The  gathering  and/or  estimating  of  cost  data  was  seen  to  be  an 
important  and  difficult  problem,  but  not  nearly  as  critical  a  problem  as  that  of  estimating  the 
effectiveness,  and  hence  the  value,  of  military  training.  Through  the  case  study  we 
indicated  the  types  of  data  that  are  necessary  and  suggested  the  requirements  for  properly 
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combining  them.  However,  it  was  necessary  to  stop  short  of  concluding  exactly  how  to 
gather  and  combine  the  necessary  data.  In  most  real-world  cases,  this  process  would 
necessarily  be  ad  hoc  and  based  on  a  set  of  guidelines.  It  would  depend  on  what  data  were 
currently  available  and  what  new  data  might  be  gathered  within  the  time  frame  of  interest 

In  both  versions  of  our  CBI/TVD  electronics  maintenance  training  case  study,  we 
saw  that  the  actual  gains  in  training  effectiveness  for  our  investment  were  signiHcant.  In 
the  first  version  of  our  hypothetical  case  study,  we  found  a  cost  decrease  of  15.16  percent, 
a  training  effectiveness  increase  of  6.66  percent,  and  a  cost-effectiveness  increase  of 
25.75  percent.  In  the  second  version,  we  found  a  cost  decrease  of  22.97  percent,  no 
change  in  training  effectiveness  (per  our  arbitrary  assumption  of  training  to  criterion),  and  a 
29.83  percent  increase  in  cost  effectiveness  (a  15.84  percent  greater  increase  in  cost- 
effectiveness  than  that  for  Version  1). 

In  both  cases,  the  recommendation  was  to  move  forward  with  the  expenditure,  not 
only  because  of  the  anticipated  gains  in  training  effectiveness,  but  also  because  of  the  value 
of  new  technology  insertion  and  other  somewhat  intangible  and  unmeasurable  aspects  of 
the  new  training  technology.  An  important  aspect  of  the  new  CBI/TVD  technology  used  in 
these  case  studies  is  its  closer  approximation  to  the  real-world  setting  in  which  actual 
electronics  mdntenance  activities  will  occur.  The  reader  is  cautioned  that  these  findings  are 
only  representative  of  what  could  be  expected  from  such  a  calculation.  Their  purpose  is 
only  to  demonstrate  the  operation  and  utility  of  the  TFDM  and  should  not  be  used  to 
promote  the  specific  use  of  CBI/IVD.  While  our  findings  are  likely  to  be  comparable  to 
real  data  on  CBI/IVD  utility,  they  are  based  only  on  estimates  derived  from  the  literature 
and  some  rather  arbitrary  cost  assumptions  made  for  clarity  of  presentation. 

B.  CONCLUSIONS  AND  RECOMMENDATIONS 

The  fundamental  utility  of  the  TFDM  concept  lies  in  its  inherent  objectivity  and 
reliability.  The  TFDM  will  provide  the  training  policy-maker  with  a  systematic,  auditable 
methodology  that  is  comprehensive  enough  to  serve  his  current  functions  and  flexible 
enough  to  accommodate  foreseeable  changes  in  the  planning,  programming,  and  budgeting 
cycle.  Instead  of  making  purely  experience-based  decisions  concerning  the  type,  amount, 
medium,  and  mix  of  training  that  is  best  for  the  military  today,  the  training  policy-maker 
will  have  a  set  of  flexible,  comprehensive  heuristics  with  which  to  guide  his  decision 
processes.  These  heuristics  will  also  promote  the  development  of  justifications  which  are 
directly  derived  from  the  cost  and  effectiveness  data  accessible  from  today’s  existing 


training  cost  and  effectiveness  data  bases.  In  the  event  that  the  required  data  do  not  exist, 
the  TFDM  would  assist  in  the  identiHcation  of  the  characteristics  associated  with  the 
necessary  information.  In  theory,  justifications  based  on  a  rigorous  logic  trail  and  reliable 
data  would  be  more  defensible  than  those  based  on  subjective  preferences.  In  accordance 
with  this  position  we  recommend  that  ASD(FM&P)TP: 

•  Continue  to  develop  and  refine  the  TFDM,  moving  toward  system  design  and 
implementation. 

•  Begin  the  development  of  TFDM  data  bases  using  past  cost-effectiveness 
analyses. 

•  Perform  one  or  more  additional  detailed  case  studies  to  validate  TFDM  and 
associated  data  base  design  and  development  concepts. 

•  Establish  an  organization  within  OASD(FM&P)TP  to  oversee  and  guide  the 
collection  of  the  data  necessary  to  conduct  military  training  cost-effectiveness 
analyses. 
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Table  1.  A  Comprehensive  Cost  Element  Structure  for  Military 
Training  Programs,  Courses,  and  Devices 


A.  RESEARCH  AND  DEVELOPMENT 

1 .  Design 

a.  Pay  and  Allowances 

(1)  Military 

(2)  Civilian 

b.  Other  Government  Personnel  Costs 

(1)  Military 

(2)  Civilian 

c.  Other 


2.  Component  Development 

a  Pay  and  Allowances 

(1)  Military 

(2)  Civilian 

b.  Other  Government  Personnel  Costs 

(1)  Military 

(2)  Civilian 

c.  Other 


3.  Producibility  Engineering  and  Planning 

a  Pay  and  Allowances 

(1)  Military 

(2)  Civilian 

b.  Other  Government  Personnel  Costs 

(1)  Military 

(2)  Civilian 

c.  Other 


(continued) 
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Tooling 


Table  1.  (continued) 


a  Pay  and  Allowances 

(1)  Military 

(2)  Civilian 

b.  Other  Government  Personnel  Costs 

(1)  Military 

(2)  Civilian 

c.  Other 


Prototype  Manufacturing 

a.  Pay  and  Allowances 

(1)  Military 

(2)  Civilian 

b.  Other  Government  Personnel  Costs 

(1)  Military 

(2)  Civiiian 

c.  Other 


Data 

a.  Managerial  Data 

(1)  Pay  and  Allowances 
(£0  Military 

(b)  Civilian 

(2)  Other  Government  Personnel  Costs 

(a)  Military 

(b)  Civilian 

(3)  Other 

b.  Technical  Data 

(1)  Pay  and  Allowances 

(a)  Military 

(b)  Civilian 

(2)  Other  Government  Personnel  Costs 

(a)  Military 

(b)  Civilian 

(3)  Other 


(continued) 


Table  1.  (continued) 

7.  P/C/D  Test  and  Evaluation 

a.  Pay  and  Allowances 

(1)  Military 

(2)  Civiiian 

b.  Other  Government  Personnel  Costs 

(1)  Military 

(2)  Civiiian 

c.  Other 


8.  System/Project  Management 

a.  Pay  and  Allowances 

(1)  Military 

(2)  Civiiian 

b.  Other  Government  Personnel  Costs 

(1)  Military 

(2)  Civilian 

c.  Other 


9.  Facirities 


10.  Other 


iNiTIAL  INVESTMENT 
1 .  Production 

a  Nonrecurring 

(1)  Production  Pianning 

(2)  Production  Tooling  and  Equipment 

(3)  Industrial  Facilities 

(4)  Other 

b.  Recurring 

(1)  Manufacturing 

(2)  Sustaining  Engineering 

(3)  Sustaining  Tooling 

(4)  Quality  Assurance 

(5)  Other 

c.  initial  Spares  and  Repair  Parts 

(continued) 


Table  1.  (continued) 

2.  Engineering  Changes 


3.  Purchased  P/C/D--Peculiar  Equipment 


4.  Common  Equipment 


5.  Data 

a  Managerial  Data 

(1)  Pay  and  Allowances 

(2)  Other  Government  Personnel  Costs 

(3)  Other 

b.  Technical  Data 

(1)  Pay  and  Allowances 

(2)  Other  Government  Personnel  Costs 

(3)  Other 

c.  Instructional  Materials 

(1)  Pay  and  Allowances 

(2)  Other  Government  Personnel  Costs 

(3)  Other 

6.  Training  P/C/D  Test  and  Evaluation 
a  Pay  and  Allowances 

b.  Other  Government  Personnel  Costs 

c.  Other 

7.  System/Project  Management 

a.  Pay  and  Allowances 

b.  Other  Government  Personnel  Costs 

c.  Other 

8.  Rents 

9.  Operational/Site  Activation 


(continued) 
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Table  1.  (continued) 


10.  initial  Training 
a  Instructors 

b.  Maintenance  Personnel 

11.  Transportation 

a  First-Destination 
b.  Second-Destination 


12.  Other 


C.  OPERATING  AND  SUPPORT 
1 .  Direct  Costs 

a.  Instructional  Costs 

(1)  Pay  and  Allowances 
(2^  Instructors 

(0  Military 
(iO  Civilian 

(b)  Supervisors,  Administrators  and  Support  Personnel 

(i)  Military 

(ii)  Civilian 

(c)  Maintenance  Personnel 

(i)  Military 

(ii)  Civilian 

(2)  Other  Government  Personnel  Costs 

(a)  Military 

(b)  Civilian 

(3)  Consumption 

(a)  POL 

(b)  Training  Munitions 

(c)  Utilities 

(i)  Electric  Power 

(ii)  Other 

(d)  Instmctional  Materials 

(e)  Other 

(4)  Replenishment  Spares 

(5)  Modification  Materiel 

(6)  Depot  Maintenance 

(a)  Labor  and  Materials 

(b)  Second  Destination  Transportation 

(c)  Other 

(7)  Other  Purchased  Services 

(8)  Other 


(continued) 


Table  1.  (continued) 

b .  T raining  Activity  Costs 

(1)  Pay  and  Allowances 

(a)  Military 

(b)  Civilian 

(2)  Other  Government  Personnel  Costs 
(sO  Military 

(b)  Civilian 

(3)  Other 

(a)  Consumables 

(b)  Other 

c.  Airfield  and  Carrier  Operations  Costs 

(1)  Pay  and  Allowances 
(sO  Military 

(b)  Civilian 

(2)  Other  Government  Personnel  Costs 

(a)  Military 

(b)  Civilian 

(3)  Other 

d.  Student  Costs 

(1)  Pay  and  Allowances 

(a)  Military 

(b)  Civilian 

(2)  Other  Student  Costs 

(a)  Military 

(b)  Civilian 

(3)  Other  Direct  Costs 


Indirect  Costs 

a.  Base  Operations 

(1)  Pay  and  Allowances 

(a)  Military 

(b)  Civilian 

(2)  Other  Government  Personnel  Costs 

(a)  Military 

(b)  Civilian 

(3)  Other 

b.  Inventory  and  Supply  Management 

(1)  Pay  and  Allowances 

(a)  Military 

(b)  Civilian 

(2)  Other  Government  Personnel  Costs 

(a)  Military 

(b)  Civilian 

(3)  Other 


(continued) 
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Military  Family  Housing  Support 

(1)  Pay  and  Allowances 
(£0  Military 

(b)  Civilian 

(2)  Other  Government  Personnel  Costs 
(cO  Military 

(b)  Civilian 

(3)  Other 

Command  Support  Costs 

(1)  Pay  and  Allowances 

(a)  Military 

(b)  Civilian 

(2)  Other  Government  Personnel  Costs 

(a)  Military 

(b)  Civilian 

(3)  Other 


Other  Indirect  Costs 
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STEPS  TO  BE  TAKEN  AND  ASSOCIATED  QUESTIONS  TO 
BE  ASKED  IN  MAKING  TRAINING-POLICY  DECISIONS 

This  brief  document,  in  outline  fonn,  overviews  5  key  steps  to  be  taken  and 
associated  questions  to  be  asked  and  answered  in  the  course  of  making  training-policy 
decisions.  Implicit  in  each  step  is  a  set  of  data  requirements  which  must  be  fulfilled.  The  5 
major  steps  are  as  follows: 

I .  Make  Policy  Decision  Explicit 

II.  Make  the  Training  Options  Explicit 

III.  Identify/Assess  the  Costs  of  the  Opticms 

rv.  Identify/Assess  the  Benefits  (Training  Effectivenesses)  of  the  Options 
V.  Perform  a  Cost-Effectiveness  Analysis  of  the  Options. 

The  elements  of  these  steps  and  associated  questions  are  discussed,  in  turn,  below. 

I.  MAKE  THE  POLICY  DECISION  EXPLICIT  BY  CLEARLY  DEFINING 
THE  DECISION  CONTEXT-THAT  IS,  FRAME  THE  TRAINING- 
POLICY  DECISION 

There  are  at  least  1 1  possible  questions  to  be  asked  in  attempting  to  clearly  define 
the  decision  context: 
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A.  Is/Are  New  or  Existing  Method(s)  of  Instruction  Involved? 

B .  Are  We  Concerned  With  One  Training  or  Operational  Resource  or  a 
Mix? 

C.  Are  We  Concerned  With  Individual,  Unit,  and/or  Command 
Performance? 

D.  Is  Our  Focus  on  Technology,  Individual  Courses,  and/or  a  Broad 
Specialization  Area  of  Training  (e.g..  Maintenance)? 

E .  Are  We  Seeking  the  Best  Mix  of  Training  and  Operational  Resources? 

F.  Are  We  Interested  in  the  Ease  of  Training  for  a  New  System? 

G.  Are  We  Focusing  on  the  Development,  Implementation,  and/or 
Evaluation  of  a  Method? 

H.  Is  Our  Focus  on  Tasks  and  Associated  Performance  Objectives  (i.e., 
OTJ  Requirements)? 

I .  Is  Our  Focus  the  Evaluation  of  a  Training  Organization  and 
Infrastructure  Versus  Individual  Courses? 

J.  Are  We  Focused  on  the  Analysis  of  Training  Research  Versus  Training 
per  se? 

K.  Do  Our  Decisions  Concern  Training-Related  System  Design  Versus 
Training  per  se? 

NOTE:  This  set  of  questions  is  not  meant  to  be  all-inclusive.  Lessons  learned  from 

previous  similar  analyses,  especially  with  respect  to  appropriate  options,  benefits,  and 

costs  to  consider,  are  critical  to  proper  and  complete  definition  of  the  decision  context. 

II.  MAKE  THE  TRAINING  OPTIONS  EXPLICIT 

The  following  potential  training  options  must  be  identified  relative  to  the  current 

decision: 


A.  Current  Training  Regimen 


B .  All  Realistically  Appropriate  Optional  Training  Regimens 

C.  Possible  Future  Alternative  Training  Regimens  Resulting  From 
Successful  Research  and  Development 

To  be  fully  specified,  we  must  have  answers  to  at  least  the  following  7  questions 
for  each  option: 

1 .  What  training  type  is  involved? 

2 .  What  are  the  performance  standards  to  be  achieved? 

3 .  What  training  time  and/or  frequency  is/are  involved? 

4 .  What  will  the  training  volume  be? 

5 .  What  training  technique(s)  will  be  involved? 

6 .  What  is/are  the  specific  course  design(s)? 

7 .  What  is  the  specific  location  of  training? 

III.  IDENTIFY/ASSESS  THE  COSTS  OF  EACH  OPTION 

A.  Precisely  Focus  the  Cost  Estimates 

1 .  What  MOS(s)  are  involved? 

2 .  What  rank(s)  are  involved? 

3 .  What  skills  are  involved? 

4 .  What  specific  course  ot  course  sequence  is  involved? 

B .  Determine  the  Average  Cost  to  Provide  Each  Training  Option 

1 .  What  is  the  long  run  average  cost  to  train  an  individual  (i.e.,  simple  formal 
training  costs  taking  student  attrition  into  account)? 

a.  What  are  the  research  and  development  costs? 

b.  What  are  the  initial  investment  costs? 

c.  What  are  the  direct  operating  costs? 

d.  What  are  the  support  costs? 
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2 .  What  costs  must  minimally  be  included  in  the  direct  operating  costs? 

a.  What  are  the  student  costs  (taking  time  to  train  and  wage  rate  into 
account)? 

b .  What  are  the  travel,  food,  and  lodging  costs? 

c .  What  are  the  instructor  costs? 

d.  What  are  the  equipment  costs  and  facilities  costs? 

e.  What  are  the  other  related  costs? 

3 .  What  special  costs  must  be  accounted  for  in  the  case  of  initial  applications  of  a 
new  technology? 

a.  What  is  the  cost  of  adapting  the  technology  to  a  new  application  (beyond 
R&D)? 

b.  Are  there  are  any  nonstandard/specialized  hardware,  software,  and/or 
courseware  costs? 

4.  To  what  cost  savings  resulting  from  reduced  time  to  train  and  increased 
information  retention  must  we  especially  attend? 

a.  Are  there  reduced  costs  for  an  individual? 

b.  Are  there  reduced  costs  for  the  total  training  infrastructure  due  to  higher 
throughput? 

IV.  IDENTIFY/ASSESS  THE  BENEFITS  (TRAINING  EFFECTIVENESS) 
OF  EACH  OPTION 

A.  Identify  the  Level  at  Which  Training  Effectiveness  Is  To  Be  Assessed 

1 .  Is  our  focus  the  individual  attainment  of  specific  performance  objectives  at  the 
end  of  the  training  option? 

2.  Is  our  focus  individual  performance  within  the  context  of  the  unit  (transfer  of 
training)? 

3 .  Is  our  focus  the  unit's  ability  to  accomplish  its  mission? 

4 .  Is  our  focus  the  command's  ability  to  accomplish  its  mission? 

B .  Measure  Training  Effectiveness  (Proficiency/Ability  Increments) 

1 .  What  are  the  values  of  certain  organizational  measures— training  time,  attrition 
rates,  and  recycle  rates? 


2.  What  are  the  values  of  certain  personnel  measures—test  scores  (pre-  and  post-), 
work  sample  evaluations  (pre-  and  post-),  measures  of  satisfaction,  and  fitness 
reports  (pre-  and  post-)? 

3.  What  are  the  values  of  certain  instructional  parameters-range  of  knowledge 
and  performance  skills  taught  and  then  measured  by  the  assessment  procedure? 

C .  Make  Use  of  Data  Bases  and  the  Research  Literature  as  Necessary  and 
Appropriate 

1 .  Are  there  any  relevant  previous  applications  of  the  same  technology? 

2 .  Are  there  any  relevant  previous  applications  of  similar  technologies? 

3 .  Are  data  available  on  key  issues  such  as  time  to  train  to  criterion,  resilience  of 
learned  skills  to  decay  over  time,  value  of  training  beyond  criterion  level? 

D.  Do  Not  Neglect  Certain  Difficult>to-Measure  Benefits  That  Might 
Result  From  an  Option 

1 .  Is  there  greater  realism  in  training? 

2 .  Is  the  instruction  more  precisely  tailored? 

3.  Is  there  qualitatively  different  training  integral  to  new  technoogy,  e.g., 
performance  of  actions  that  would  wdinarily  be  harmful  to  trainees  and/or  their 
equipment? 

V .  PERFORM  A  COST-EFFECTIVENESS  ANALYSIS  OF  THE  OPTIONS 

A.  Compute  the  Simple  Formal  Training  Costs  for  Each  Option 
(see  HI,  above) 

Key  questions  to  be  asked  concerning  cost  elements  are  the  following: 

1 .  What  is  the  cost  impact  of  any  decrease  in  student  training  time  from  the  new 
technology  (from  the  research  literature)? 

2.  What  is  the  cost  impact  of  any  decrease  in  instructor  time  resulting  from  the 
new  technology  (estimated  using  the  research  literature)? 
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B .  Convert  the  Estimates  of  Training  En'ectiveness  of  Each  Option  to 
Common  Interval  Scale  Measures  of  Student  Average  ProHciency/ 
Ability  at  Performing  the  Requisite  Tasks/Duties/Skills 

(see  IV,  above) 

1 .  What  are  the  values  of  certain  measures  of  student  average  proficiency/ability 
gains  at  performing  the  requisite  tasks/duties/skills  for  each  option,  taking 
speed  and  accuracy  into  account? 

a.  What  are  the  data  on  average  training  effect  size  due  to  the  alternative 
training  regimens  (from  the  research  literature  if  necessary)  for  the 
appropriate  student  population  and  performance  area? 

(i)  Do  we  have  knowledge  and  performance  training  outcomes  (the 
results  from  paper  tests  and  performance  tests)? 

(ii)  Do  we  have  data  on  post-training  retention  of  trained  skills  and 
learned  knowledge? 

(iii)  Do  we  have  post-training  fitness  reports  (or  estimates  of  OTJ 
performance)  for  students  subjected  to  each  training  option? 

(iv)  Do  we  have  measures  of  time  on  task  during  training  (secondary 
measure)? 

(v)  Do  we  have  measures  of  time  to  complete  instruction  (secondary 
measure)? 

(vi)  Do  we  have  measures  of  time  to  complete  performance/knowledge 
tests  (secondary  measure)? 

2 .  What  is  an  appropriate  function  for  converting  the  various  measures  of  student 
proficiency/ability  gains  to  an  interval  scale  (if  there  is  not  one  already)? 

3 .  What  is  an  appropriate  function  for  converting  training  effect  sizes  to  interval 
scale  estimates  of  military  value  of  training  (readiness?)?  [Optional] 

C.  Calculate  Ratios  of  Effectiveness  Divided  by  Cost 

D.  Perform  Comparisons  of  Effectiveness/Cost  Ratios  and  Other  Key 
Factors 

1.  What  are  the  percent  difference(s)  in  cost-effectiveness  between/among 
options? 

2.  How  do  the  number  of  trainees  through  the  pipeline  per  unit  time  compare? 

3 .  How  do  the  forcewide  increases  in  proficiency  compare? 

4.  How  do  the  total  costs  of  training  for  all  trainees  compare? 
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APPENDIX  C 

INFORMATION  ITEMS  FOR  TRAINING  POLICY  DECISIONS 


FIVE  KEY  GROUPINGS  FOR  INFORMATION  ITEMS 

I .  The  Explicit  Policy  Decision  to  be  Addressed 

II .  The  Explicit  Training  Options  Involved 

III.  The  Costs  of  the  Options 

rv.  The  Benefits  (Training  Effectivenesses)  of  the  Options 
V.  Cost-Effectiveness  Measures  for  the  Options 

I,  DECISION  CONTEXT  INFORMATION  ITEMS 

1 .  New  method(s)  of  instruction  involved 

2 .  Existing  inethod(s)  of  instractitxi  involved 

3.  Training  or  operational  resource(s)  involved 

4.  Specific  individual,  unit,  or  command  performance  knowledge/skills  to  be 
measured 

5.  Technology  addressed 

6.  Individual  course(s)  involved 

7 .  Broad  specializatitm  area  of  training  (e.g.,  maintenance)  involved 

8 .  New  system  for  which  training  is  to  be  provided 

9 .  Training  method  to  be  develcq}ed,  implemented,  or  evaluated 

10.  Tasks  and  associated  performance  objectives  (i.e.,  OTJ  requirements)  to  be 
addressed 

11.  Training  organization  and  infirastructure  to  be  evaluated 

12.  Training  research  to  be  analyzed 

1 3.  Training-related  syston  design  decisions  to  be  made 
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II.  INFORMATION  ITEMS  FOR  THE  TRAINING  OPTIONS 

1 .  Current  traming  regimen 

2 .  Realistically  appropriate  optional  training  regimens 

3.  Possible  future  alternative  training  regimens  given  successful  research  and 
development 

4.  Training  type 

5 .  Performance  standards  to  be  achieved 

6 .  Training  time  and/or  frequency 

7.  Training  volume 

8.  Training  technique(s)  involved 

9.  Specific  course  design(s) 

10.  Location  of  training 

III.  INFORMATION  ITEMS  ADDRESSING  THE  COST  OF  EACH 
OPTION 

1 .  MOS(s)  involved 

2.  Rank(s)  involved 

3.  Skills  involved 

4 .  Specific  course  ot  course  sequence  involved 

5.  The  long  run  average  cost  to  train  an  individual  (i.e.,  simple  formal  training 
costs  taking  student  attrition  into  account) 

6 .  Research  and  development  costs 

7 .  Initial  investment  costs 

8.  Direct  operating  costs 

9.  Support  costs 

1 0.  Student  costs  (time  to  train  and  wa^  rate) 

1 1 .  Travel,  food,  and  lodging  costs 

12.  Instructor  costs 

1 3 .  Equipment  costs  and  facilities  costs 

14.  Other  related  costs 

15.  Cost  of  adapting  technology  to  a  tww  application  (beyond  R&D) 


16.  Nonstandaid/specialized  hardware,  software,  and  courseware  costs 

17.  Cost  savings  resulting  from  reduced  time  to  train  and  increased  information 
retention  including  reduced  costs  for  an  individual  and  reduced  costs  for  total 
training  infrastructure  due  to  higher  throughput 

IV.  INFORMATION  ITEMS  ADDRESSING  THE  BENEFITS  (TRAINING 
EFFECTIVENESS)  OF  EACH  OPTION 

1 .  The  level  at  which  training  effectiveness  is  to  be  assessed 

2 .  Specific  performance  objectives  to  be  met  by  the  individual  trainee  at  the  end  of 
the  training  option 

3.  Individual  performance  objectives  within  the  context  of  the  unit  (transfer  of 
training) 

4 .  Perfonnance  objectives  with  respect  to  unit  ability  to  accomplish  its  mission 

5.  Performance  objectives  with  respect  to  command  ability  to  accomplish  its 
mission 

6 .  Training  effectiveness  (proficiency/ability  increments)  measures 

7 .  Organizational  measures— training  time,  attrition  rates,  recycle  rates 

8.  Personnel  measures-test  scores  (pre-  and  post-),  work  sample  evaluations 
(pre-  and  post-),  measures  of  satisfaction,  fimess  reports  (pre-  and  post-) 

9.  Instructional  parameters-range  of  knowledge  and  performance  skills  taught 
and  then  measured  by  the  assessment  procedure 

10.  Relevant  data  bases  and  research  literature 

1 1 .  Previous  applications  of  the  same  technology 

1 2.  Previous  applications  of  similar  technologies 

13.  Time  to  train  to  ciiteiion 

1 4.  Resilience  of  learned  skills  to  decay  over  time 

1 5 .  Value  of  training  beyond  criterion  level 

16.  Certain  difficult- to-measure  benefrts  that  might  result  from  greater  realism  in 
training,  more  precisely  tailoed  instruction,  and  qualitatively  different  training 
integral  to  a  new  technology  (e.g.,  performance  of  actions  that  would 
ordinarily  be  harmful  to  trainees  or  their  equipment) 


V .  INFORMATION  ITEMS  ADDRESSING  THE  PERFORMANCE  OF  A 
COST-EFFECTIVENESS  ANALYSIS  OF  THE  OPTIONS 

1 .  The  simple  formal  training  costs  for  each  option  (see  HI,  above) 

2.  Cost  impact  of  decrease  in  student  training  time  to  train  resulting  from  the  new 
technology  (from  the  research  literature) 

3.  Cost  impact  of  decrease  in  instructor  time  resulting  from  the  new  technology 
(estimated  from  the  research  literature) 

4.  Measures  of  student  average  proficiency/ability  gains  in  performing  the 
requisite  tasks/duties/skills  for  each  option,  taking  speed  and  accuracy  into 
account  (see  IV,  above) 

5 .  Data  on  average  training  effect  size  due  to  the  alternative  training  regimen  (from 
the  research  literature)  for  the  appropriate  student  population  and  performance 
area 

6.  Function  for  converting  measures  of  student  proficiency /ability  gains  to  an 
interval  scale  (if  not  already  on  one) 

7.  Function  for  converting  training  effect  sizes  to  interval  scale  estimates  of 
military  value  of  training  (readiness?)  [Optional] 

8.  Post-training  fitness  reports  (or  estimates)  for  students  subjected  to  each 
training  option 

9.  Knowledge  and  performance  training  outcomes  (paper  tests  and  performance 
tests) 

10.  Retention  of  trained  skills  and  learned  knowledge 

1 1 .  Time  on  task  during  training  (secondary  measure) 

1 2.  Time  to  complete  instruction  (secondary  measure) 

1 3.  Time  to  complete  performancc/knowledge  tests  (secondary  measure) 

1 4.  Ratios  of  effectiveness  divided  by  cost 

1 5 .  Comparisons  of  effectiveness/cost  ratios 

1 6.  Number  of  trainees  through  the  pipeline  per  unit  time  for  each  option 

1 7 .  Forcewide  increase  in  proficiency  for  each  option 

1 8.  Total  cost  of  training  all  trainees  for  each  option 


